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JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

1 .This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] It is the image data-conversion approach changed into the image data which expresses a 
dynamic image using image data including the color information on a subject-copy image that plurality 
continued. Generate the discontinuity information on the subject-copy image which expresses the 
discontinuity of the color information expressed according to the Rhine process based on the image data 
of a subject-copy image, and while extracting the profile of the body image contained in the subject- 
copy image based on the generated discontinuity information The image data-conversion approach 
which searches for the texture information which expresses a body image using the profile extracted in 
quest of the outline information which outline-ized the profile of a body image based on said 
discontinuity information, and encodes image data including said outline information and texture 
information. 

[Claim 2] It is the image data-conversion approach according to claim 1 characterized by outline-izing 
the profile of said body image with a parametric curve, and encoding said color information on a 
parametric curved surface. 

[Claim 3] Said discontinuity information is the image data-conversion approach according to claim 1 or 
2 characterized by generating the periphery discontinuity information equivalent to the profile of the 
body image contained in the subject-copy image, and the internal discontinuity information inside a 
body image as discontinuity information on a subject-copy image. 

[Claim 4] Among the subject-copy images with which said plurality continued, define the subject-copy 
image of criteria and it carries out [ aforementioned ] coding about the subject-copy image of the 
defined criteria. The vector information showing fluctuation of the body image contained in the subject- 
copy image of criteria about other subject-copy images is searched for. The image data-conversion 
approach given in any 1 term of claim 1 characterized by changing into the image data showing a 
dynamic image from the image data and vector information which were encoded about the subject-copy 
image of criteria thru/or claim 3. 

[Claim 5] It is the record medium which recorded the image data-conversion program for changing into 
the image data which expresses a dynamic image using image data including the color information on a 
subject-copy image that plurality continued, by computer. Said image data-conversion program is based 
on the image data of a subject-copy image including color information. While making die profile of the 
body image contained in the subject-copy image based on the discontinuity information which was 
made to generate the discontinuity information on the subject-copy image showing the discontinuity of 
the color information expressed according to the Rhine process, and was generated extract The outline 
information which outline-ized the profile of a body image based on said discontinuity information is 
made to be searched for. The record medium which recorded the image data-conversion program 
characterized by making the texture information which expresses a body image using the extracted 
profile searched for, and making image data including said outline information and texture information 
encode. 

[Claim 6] It is the record medium which the profile of said body image was outline-ized with the 
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parametric curve, and recorded the image data-conversion program according to claim 5 characterized 
by encoding said color information on a parametric curved surface. 

[Claim 7] Said discontinuity information is the record medium which recorded the image data- 
conversion program according to claim 5 or 6 characterized by making the periphery discontinuity 
information equivalent to the profile of the body image contained in the subject-copy image, and the 
internal discontinuity information inside a body image generate as discontinuity information on a 
subject-copy image. 

[Claim 8] Coding is carried out [ aforementioned ] about the subject-copy image of the criteria which 
were made to define the subject-copy image of criteria among the subject-copy images with which said 
plurality continued, and were defined. The vector information showing fluctuation of the body image 
contained in the subject-copy image of criteria about other subject-copy images is made to be searched 
for. The record medium which recorded the image data-conversion program of a publication on any 1 
term of claim 5 characterized by making it change into the image data showing a dynamic image from 
the image data and vector information which were encoded about the subject-copy image of criteria 
thru/or claim 7. 



[Translation done.] 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention starts the image data-conversion approach and the record medium 
which recorded the image data-conversion program, and relates to the record medium which recorded 
the image data-conversion approach and image data-conversion program which are changed into the 
image data which enables presentation of a dynamic image especially using two or more subject-copy 
images. 
[0002] 

[Description of the Prior Art] In recent years, with development of computer technology, a color picture 
is digitized as image data, and the image data may be made to save, or it may be made to circulate and 
may use. There are image data showing the photography image generated by taking a photograph with 
digital photography equipments, such as a digital camera, image data showing the scanning image 
generated by scanning a color copy with a scanner, etc. as this digitized image data. 
[0003] By the way, it is known by showing two or more continuous static images for every fixed time 
amount that a dynamic image can be shown. For this reason, if the image data which continuous 
plurality digitized is shown for every fixed time amount, the digitized dynamic image can be shown. 
[0004] However, since the data volume which one still picture occupies is large when a color picture is 
digitized as image data, data volume increases according to the number of the still picture shown as a 
dynamic image. For this reason, mass storage must be prepared. Moreover, in order to show each 
digitized still picture as an image, the image transformation for presentation is required for every image, 
and in order to show as a dynamic image, it is necessary to perform in the real time. For this reason, 
equipment will become large-scale and high-speed processing will be required. 
[0005] In order to solve this problem, there is the so-called MPEG specification, the difference of the 
still picture which continues considering the first still picture as criteria (criteria still picture) by MPEG 
specification about the dynamic image which consists of two or more still pictures etc. — asking — 
difference ~ a value etc. is replaced with a continuous still picture and made to memorize By making it 
this **, storage capacity can be compressed and dynamic-image presentation in the real time is enabled. 
[0006] 

[Problem(s) to be Solved by the Invention] However, although it is effective in the still picture 
approximated in expansion by the difference of a continuous still picture, it is unsuitable, when big 
actuation is included within an image or a different element newly inserts. Moreover, when 
accompanied by resolution modification of enlarging or contracting etc., sense of incongruity may arise. 
[0007] It is the purpose to obtain the image data-conversion approach with possible this invention 
reproducing the dynamic image by two or more sequential images without sense of incongruity in 
consideration of the above-mentioned fact and the record medium which recorded the image data- 
conversion program. 
[0008] 

[Means for Solving the Problem] In order to attain the above-mentioned purpose the image data- 
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conversion approach of invention according to claim 1 It is the image data-conversion approach changed 
into the image data which expresses a dynamic image using image data including the color information 
on a subject-copy image that plurality continued. Generate the discontinuity information on the subject- 
copy image which expresses the discontinuity of the color information expressed according to the Rhine 
process based on the image data of a subject-copy image, and while extracting the profile of the body 
image contained in the subject-copy image based on the generated discontinuity information The texture 
information which expresses a body image using the profile extracted in quest of the outline information 
which outline-ized the profile of a body image based on said discontinuity information is searched for, 
and image data including said outline information and texture information is encoded. 
[0009] Invention according to claim 2 is the image data-conversion approach according to claim 1, and 
it is characterized by outline-izing the profile of said body image with a parametric curve, and encoding 
said color information on a parametric curved surface. 

[0010] Invention according to claim 3 is the image data-conversion approach according to claim 1 or 2, 
and said discontinuity information is characterized by generating the periphery discontinuity information 
equivalent to the profile of the body image contained in the subject-copy image, and the internal 
discontinuity information inside a body image as discontinuity information on a subject-copy image. 
[001 1] Invention according to claim 4 is the image data-conversion approach given in any 1 term of 
claim 1 thru/or claim 3. Among the subject-copy images with which said plurality continued, define the 
subject-copy image of criteria and it carries out [ aforementioned ] coding about the subject-copy image 
of the defined criteria. The vector information showing fluctuation of the body image contained in the 
subject-copy image of criteria about other subject-copy images is searched for, and it is characterized by 
changing into the image data showing a dynamic image from the image data and vector information 
which were encoded about the subject-copy image of criteria. 

[0012] Invention according to claim 5 is the record medium which recorded the image data-conversion 
program for changing into the image data which expresses a dynamic image using image data including 
the color information on a subject-copy image that plurality continued, by computer. Said image data- 
conversion program is based on the image data of a subject-copy image including color information. 
While making the profile of the body image contained in the subject-copy image based on the 
discontinuity information which was made to generate the discontinuity information on the subject-copy 
image showing the discontinuity of the color information expressed according to the Rhine process, and 
was generated extract The texture information which expresses a body image using the profile which 
made search for it and extracted the outline information which outline-ized the profile of a body image 
based on said discontinuity information is made to be searched for, and it is characterized by making 
image data including said outline information and texture information encode. 

[0013] Invention according to claim 6 is the record medium which recorded the image data-conversion 
program according to claim 5, and it is characterized by outline-izing the profile of said body image with 
a parametric curve, and encoding said color information on a parametric curved surface. 
[0014] Invention according to claim 7 is the record medium which recorded the image data-conversion 
program according to claim 5 or 6, and said discontinuity information is characterized by making the 
periphery discontinuity information equivalent to the profile of the body image contained in the subject- 
copy image, and the internal discontinuity information inside a body image generate as discontinuity 
information on a subject-copy image. 

[0015] Invention according to claim 8 is the record medium which recorded the image data-conversion 
program of a publication on any 1 term of claim 5 thru/or claim 7. Coding is carried out 
[ aforementioned ] about the subject-copy image of the criteria which were made to define the subject- 
copy image of criteria among the subject-copy images with which said plurality continued, and were 
defined. The vector information showing fluctuation of the body image contained in the subject-copy 
image of criteria about other subject-copy images is made to be searched for, and it is characterized by 
making it change into the image data showing a dynamic image from the image data and vector 
information which were encoded about the subject-copy image of criteria. 

[0016] It changes into the image data which expresses a dynamic image with the image data-conversion 
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approach according to claim 1 using image data including the color information on a subject-copy image 
that plurality continued. Based on the color information on the subject-copy image, the discontinuity 
information on the subject-copy image showing the discontinuity of the color information expressed 
according to the Rhine process is generated. The Rhine process will have a value in the part where the 
Rhine process will become discontinuous [ color information ] by the subject-copy image if the 
discontinuity of color information, i.e., color information, expresses continuation or discontinuity, it 
defines an energy function by color information and the Rhine process and minimizes energy using an 
energy function. The discontinuity information on a subject-copy image can be expressed with the Rhine 
process which has this value. Since this discontinuity information expresses the part where color 
information becomes discontinuous by the subject-copy image, it serves as a boundary of color 
information in the adjacent color field which consists of different color information which does not 
consist of the same or similar color information. 

[0017] In addition, color information can be used as concentration information, when treating in one 
color. There is image information of a binary image as an example. Therefore, as for the color field 
which consists of the same or similar color information, discontinuity information appears in the profile 
part. The body image contained in a subject-copy image consists of the same or similar color 
information, or consists of two or more color information defined beforehand. Then, the profile of the 
body image contained in the subject-copy image based on discontinuity information is extracted. 
[0018] Next, the outline information which outline-ized the profile of a body image based on 
discontinuity information is searched for. Outline information is information in which the thing [****/ 
making magnitude change for example, expand, with most original information (namely, information 
about a profile) maintained ] made to reduce is possible. For example, there is configuration information 
which consists of a point and vector information, such as a segment and field information, among the 
outline information. In addition, description with a formula is also possible for this configuration 
information. Thus, the profile of a body image can be expressed by describing the profile of a body 
image using outline information, without being dependent on resolution. 

[0019] The endocyst of the color information on a body image is carried out to the profile of the body 
image described using outline information, and when it is the successive state without discontinuity, it 
can be encoded with the expression using continuation information. Then, the texture information which 
expresses a body image using the extracted profile is searched for. There are bit map data and an image 
data format in this texture information. Coding which carries out the approximation expression for 
example, of the color information is made into an example, and there is an image data format known for 
JPEG etc. among the image data formats. Image data including such outline information and texture 
information is encoded. 

[0020] Thus, since outline information described the profile of a body image and image data including 
the texture information on a subject-copy image or a body image is encoded, when reproducing namely, 
showing two or more subject-copy images continuously, the profile of a body image can change image 
data in the format which can be used in the format which maintained the profile of the original body 
image, without being dependent on resolution. 

[0021] In addition, while searching for outline information, you may ask for whenever [ showing the 
penetration degree when compounding a body image with other images based on discontinuity 
information / penetration ]. Without making a boundary part produce sense of incongruity by expressing 
the penetration degree of a body image about the part on the subject-copy image corresponding to 
discontinuity information, and using at the time of composition and enlarging or contracting of an 
image, enlarging or contracting of the subject-copy image can be carried out, or whenever [ this 
penetration ] can make a body image melt into other images. If this performs synthetic actuation and 
enlarging or contracting, such as sticking on other images, sense of incongruity will not arise [ the 
profile part ] on the boundary of a body image and other images. 

[0022] When expressing an image, the profile is the outermost periphery of an image and serves as a 
two-dimensional curve. Moreover, the endocyst of the color information is carried out to the profile of 
an image, and it becomes continuously. The three-dimension expression with the location and 
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concentration is possible for color information for every color. Therefore, it becomes possible to make it 
correspond to a three-dimension-curved surface of continuous color information. As for such a curve 
and a curved surface, it is desirable that handling is expressed in an easy format. 
[0023] In invention of claim 2, the profile of said body image is outline-ized with a parametric curve, 
and color information is encoded on a parametric curved surface. By doing in this way, the profile and 
color information on a body image can be expressed by the easy symbolic convention using a formula, 
and it becomes possible to use easily. 

[0024] Here, although the most exists near a periphery, it may have the description inside the body 
image, or only the interior of a body image may form delicately into a form status change the profile of 
the body image contained in the subject-copy image. For example, the description may be in parts, such 
as an eye, a nose, and opening, by the portrait image, or although the profile of a head does not carry out 
abbreviation change, there may be change of a person's expression because parts, such as an eye, a nose, 
and opening, change. 

[0025] In invention of claim 3, the periphery discontinuity information which is equivalent to the profile 
of the body image contained in the subject-copy image as said discontinuity information, and the 
internal discontinuity information inside a body image are generated as discontinuity information on a 
subject-copy image. Thus, while being able to extract the discontinuity of the profile showing the 
description of a body image, and the interior by describing the profile of a body image, and the 
discontinuity information on internal using outline information, it can express, without being dependent 
on resolution. 

[0026] By the way, when changing into the image data which expresses a dynamic image using image 
data including the color information on a subject-copy image that plurality continued, a processing load 
increases in having encoded image data including the texture information on the outline information and 
the subject-copy image showing the profile of a body image, or a body image about each subject-copy 
image. 

[0027] In invention of claim 4, the subject-copy image of criteria is defined, and is encoded about the 
subject-copy image of the defined criteria among the subject-copy images with which plurality 
continued, the vector information showing fluctuation of the body image contained in the subject-copy 
image of criteria about other subject-copy images is searched for, and it changes into the image data 
which expresses a dynamic image from the image data and vector information which were encoded 
about the subject-copy image of criteria. A body image may consist of two or more element images. For 
example, although a background does not change, only a person may move or only a person's expression 
may change. In this case, it is efficient if only the changing image is memorized. Therefore, it encodes 
about the subject-copy image of criteria, and if the vector information showing fluctuation of the body 
image contained in the subject-copy image of criteria about other subject-copy images is searched for, 
capacity of the image data showing a dynamic image can be made small. Moreover, a processing load is 
mitigated also for transform processing to the image data which expresses a dynamic image from the 
image data of the subject-copy image of criteria and vector information which were encoded. 
[0028] Coding of the subject-copy image by the above-mentioned image data-conversion approach is 
realizable on a computer with the image data-conversion program execution recorded on the record 
medium according to claim 5. It is the record medium which recorded the image data-conversion 
program for changing into the image data which expresses a dynamic image using image data including 
the color information on a subject-copy image that plurality continued, by computer on the detail. Said 
image data-conversion program is based on the image data of a subject-copy image including color 
information. While making the profile of the body image contained in the subject-copy image based on 
the discontinuity information which was made to generate the discontinuity information on the subject- 
copy image showing the discontinuity of the color information expressed according to the Rhine 
process, and was generated extract The texture information which expresses a body image using the 
profile which made search for it and extracted the outline information which outline-ized the profile of a 
body image based on said discontinuity information is made to be searched for, and image data 
including said outline information and texture information is made to encode. When setting on a 
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computer and reproducing namely, showing two or more subject-copy images continuously by this, the 
profile of a body image can change image data in the format which can be used in the format which 
maintained the profile of the original body image, without being dependent on resolution. 
[0029] in addition, it indicated also to claim 6 — as — the profile of said body image — a parametric 
curve — outline — said color information can be encoded so that it may encode on a parametric curved 
surface, so that it may be-izing and carried out. 

[0030] Moreover, said discontinuity information can make the periphery discontinuity information 
equivalent to the profile of the body image contained in the subject-copy image, and the internal 
discontinuity information inside a body image generate as discontinuity information on a subject-copy 
image, as indicated also to claim 7. 

[003 1] Furthermore, the inside of the subject-copy image with which said plurality continued as 
indicated also to claim 8, Coding is carried out [ aforementioned ] about the subject-copy image of the 
criteria which were made to define the subject-copy image of criteria and were defined. The vector 
information showing fluctuation of the body image contained in the subject-copy image of criteria about 
other subject-copy images can be made to be searched for, and it can be made to change into the image 
data showing a dynamic image from the image data and vector information which were encoded about 
the subject-copy image of criteria. 
[0032] 

[Embodiment of the Invention] Hereafter, an example of the gestalt of operation of this invention is 
explained to a detail with reference to a drawing. The gestalt of this operation applies this invention to 
the image transformation equipment which generates the image data which expresses a dynamic image 
from two or more color subject-copy images. 

[0033] As shown in drawing 2 , the image transformation equipment 10 of the gestalt of this operation 
consists of image readers 30 for reading a color subject-copy image from the input units 14, such as a 
keyboard for inputting a command and data from the display units 12, such as an indicating equipment 
for displaying an image, and the exterior, the body 16 of equipment, and the color copy. 
[0034] the body 16 of equipment - CPU18, RAM20, ROM22, and input/output port (I/O) 28 - since ~ 
it consists of becoming microcomputers, and each is connected by the bus 26 so that transfer of a 
command or data may be possible. In addition, the manipulation routine which is performed in the body 
16 of equipment and which is mentioned later is memorized by ROM22. 

[0035] The memory 24 for memorizing image data is connected to the input/output port 28 of the body 
16 of equipment. Moreover, while an input device 14 is connected, the display unit 12 is connected to 
the input/output port 28 of the body 16 of equipment. Moreover, the image readers 30, such as a color 
scanner, are connected to input/output port 28. 

[0036] In addition, the image reader 30 can read the color subject-copy image of a multiple value in 
color copies, such as printed matter, and the photographic film with which the development was carried 
out after photography of a photographic subject, and the negative image or the positive image was 
visualized. 

[0037] The floppy disk unit (henceforth FDU) in which the insert and remove of the floppy (trademark) 
disk (henceforth FD) as a record medium are possible is connected to the above-mentioned input/output 
port 28. In addition, the manipulation routine mentioned later can be written to FD using FDU. 
Therefore, without memorizing to ROM22, the manipulation routine mentioned later is beforehand 
recorded on FD, and may perform the processing program recorded on FD through FDU. Moreover, 
large capacity storage (illustration abbreviation), such as a hard disk drive unit, is connected to the body 
16 of equipment, and the processing program recorded on FD is stored in large capacity storage 
(illustration abbreviation) (install), and it may be made to perform it. Moreover, what is necessary is to 
replace with Above FDU or just to use CD-ROM equipment, MD equipment, MO equipment, etc. 
further, when there are magneto-optic disks, such as optical disks, such as CD-ROM, and MD, MO, and 
these are used as a record medium. 

[0038] In addition, although the gestalt of this operation explains as an example the case where the color 
subject-copy image of a multiple value is inputted with the image readers 30, such as a color scanner, 
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this invention is not limited to this and you may make it input the image data memorized beforehand 
into storages, such as FD. Moreover, signal communication units, such as a network board, are 
connected to input/output port 28, the so-called LAN which enables signal transfer among other 
equipments is constituted, and you may make it receive image data from other equipments. 
[0039] Moreover, the color subject-copy image of a multiple value contains images, such as an on-the- 
spot photo image and a natural image, from monochrome image or the binary image by different 
combination of a color. 

[0040] Next, an operation of the gestalt of this operation is explained. First, coding of the static image 
which is a subject-copy image is explained in image transformation. In addition, the following 
processings (coding manipulation routine) are stored in record media, such as FD, are offered as 
application which can be performed by computer, and may be made to perform with activation 
directions. 

[0041] First, a color subject-copy image (henceforth a subject-copy image) is read by reading the color 
copy laid in the image reader 30 in step 100. With the gestalt of this operation, RGB data shall be 
outputted from the image reader 30, and this outputted RGB data shall be used. In addition, when it is 
the image data expressed with other color coordinate systems like CMYK data as image data, what is 
necessary is just made to perform RGB conversion. 

[0042] An example of the subject-copy image 40 which reading ended was shown in drawing 4 . 
Flowers 50 are scattered around a person 52, a garden tree 48 is located in the near, far away (upper part 
of an image), a crest 42 and Oyama 44 are located, and, as for the subject-copy image 40, clouds 46 are 
located up. 

[0043] At the following step 102, detection of discontinuity information is made according to the Rhine 
process about a subject-copy image. The component of the imagination which shows the Rhine process 
(stroke) discontinuity is said. With the gestalt of this operation, the neural network who used the Rhine 
process explains the case where the discontinuity information on an image is detected. First. A detail is 
explained about the Rhine process. 

[0044] Drawing 5 (A) showed about the pixel an example of the 1 -dimensional data which asked for the 
relation between a location and concentration, and points P1-P6 and points P7-P12 were located near the 
abbreviation, and it is greatly separated from them of between a point P6 and points P7. When it asks by 
the approximation approaches and the interpolation approaches, such as the least square method, in 
order to make points P7-P12 continue with a curve from points P1-P6 as shown in drawing 5 (B) when 
asking for the curve which meets these 1 -dimensional data, to become smooth is asked although a curve 
34 separates from a point P6 and a point P7 greatly. 

[0045] However, since a point P6 and a point P7 will separate from a curve 34 greatly, the property 
(namely, curve 34) acquired will be **(ed) to actual data. Then, about the points P1-P6 and the points 
P7-P12 of being located near the abbreviation, as shown in drawing 5 (C), it asks for Curves 34a and 
34b so that each point may be met. Thus, if it asks, discontinuity will arise between the points P6 and 
points P7 which are greatly separated. If Curves 34a and 34b are set to the location between this point 
P6 and a point P7 with a break point 36, the property which **(ed) to actual data can be searched for. 
That is, the Rhine process showing discontinuity occurring in the break point 36 of the location between 
a point P6 and a point P7 serves as ON. moreover, each ** PI of a point PI - a point P6 and a point P7 - 
a point P12, i.e., a point, a point P2 and a point P2, a point P3, and ... it becomes off [ the Rhine 
process ] between each of a point P5, a point P6 and a point P7, a point P8 and a point P8, a point P9, ... 
and a point PI 1, and a point PI 2. 

[0046] Although the case of 1 -dimensional data was explained, when applying the Rhine process to the 
image represented with two-dimensional data above, the Rhine process is defined as an imagination 
function about between two pixels. Therefore, when the defined Rhine process turns on or turns off 
according to a local concentration difference, the thing with which the discontinuous part in an image 
made discontinuity information and this discontinuous part connect can extract as a profile of a body 
image. 

[0047] Count and its minimization of an energy function are performed by the neural network using this 
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Rhine process, and it asks for the discontinuity (information) of an image. This detail is explained. 
[0048] Drawing 6 is a conceptual block diagram for explaining the neural network who used the Rhine 
process. They are three neurone fi and j, and hi, j, vi and j to one pixel corresponding to [ as shown in 
drawing 6 , when the input of this neural network is an image ] an image. It corresponds. 
[0049] Here, they are fi and j. It is the value of a pixel and they are hi, j, vi, and j. fi and j It is the hidden 
function called the Rhine process which shows the existence of discontinuity of a between, and 
nonexistence. Namely, fi and j About a change of the value inputted on the strength, they are hi, j, vi, 
and j. They are fi and j, respectively. The continuity of horizontal and a perpendicular direction and 
discontinuity are shown. 

[0050] These three variables fi and j, hi, j, vi, and j The energy function of the whole system is defined 

as shown in the following (1) types. 

[0051] 

J Equation 1] 
i-E l *E„ + E v + E r + E c + E L + E a • • • (1) 
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[0052] however EI : [ The continuity ED of curved-surface (depth) data ] : Dependability EV of 
observation data (initial value) : Thing EP which the Rhine process tends toward the angle (Oorl) of a 
hypersphere side : Conditions EC to set to 1 the Rhine process to which near was parallel : Conditions 
EL to set the single Rhine process to 1 : The continuous multi-line process is liked. Conditions EG 
which dislike a crossover and discontinuity : The conditions CD, valve flow coefficient, and CP for m in 
(2) types and n not to emit, CC, CL, CG : Parameter- value g(): Sigmoid functions di and j : The 
following (2) types defined initial value and the time amount rule of a variable. 
[0053] 
[Equation 2] 

d/ tJ /dt=dE/df lti 1 

dm^/dt^dE/dh^ > • • • ( 2 ) 
dn tJ /dt=dE/dv tJ 

[0054] However, g(Ui) =l/e (-21ambdaUi) 
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In addition, e (-21ambdaUi) is hi and j =g (mi and j) to which it is an exponential function and the 
interior of ( ) expresses exponent part, 
vi j =g(ni j ) 

m 3 n: The example of count of the partial differential of the internal-state variable above-mentioned (2) 

type is shown in (3), (following 4), and following (5) types. 

[0055] 

[Equation 3] 



[7] ID=000005 



[0056] the value fi which the count result of the above-mentioned (3) - (5) type becoming small, i.e., 
taking the value of 0 or it is close to 0 adjoins, and j+1 Values fi and j and fi+1, and j fi and j 
abbreviation — it is a time of becoming the same value, therefore, energy EI expressed with the above- 
mentioned (1) type ******** ~ fi and j+1 ** - fi, j and fi+1, and j ** - fi and j it is - supposing - 
energy EI The Rhine processes hi and j since it becomes comparatively small, vi, and j since it is not 
necessary to have a value — hi, j, vi, and j It becomes a comparatively small value. 
[0057] The value fi and j+1 which adjoin on the other hand Values fi and j and value fi+1, and j Values 
fi and j When a boundary is between the values with a large difference which case [ values ] namely, 
adjoin, it is 2 (fi, j+1- fi, j). And (fi+ 1, j- fi and j) 2 It becomes large. Therefore, energy EI For making it 
small, they are hi and j. Or vi and j It has a value, or (1- hi and j) (1- vi and j) it is necessary to become 
comparatively small. Thus, values fi and j which adjoin mutually fi and j+1 Or fi and j fi+1 and j They 
are the Rhine processes hi and j which are between each value when there is a difference, vi, and j. It 
comes to have a value and a boundary line will appear between the fields of a different color. 
[0058] The conditions which the Rhine process generates from this multiplier are determined. The 
conditions which the single Rhine process shown in drawing 7 (A) generates as an example EP =0, EC 
=0, and EL =2CL, The conditions which show the conditions which the continuous Rhine process 
shown in drawing 7 (B) generates to EP =0, EC =0, EL =CL, and drawing 7 (C) and which the parallel 
Rhine process generates mutually the conditions which EP =CP, EC =CC, EL =2CL, and the Rhine 
process that crosses mutually as shown in drawing 7 (D) generate - EP =0, EC =2cc, and EL =10CL it 
is . 

[0059] The result of having performed energy study which repeats the count which makes energy of the 
whole system min by the above-mentioned (1) - (5) type serves as a solution over the given input to. 
[0060] This neural network is (a). When an input is the image to which the noise was added, fi in the 
obtained energy min, and j It is equivalent to a restoration image and is (b). When an input is a lightness 
image, hij in the obtained energy min and vij are equivalent to a profile, and are (c). In the case of 
geodetic datas, such as a crest, an input will show the altitude of the every place point that fij in the 
obtained energy min is presumed from a surveying point. At the gestalt of this operation, it is (b). It is 
the example used for the image input. 

[0061] In addition, since the above-mentioned neural network has expandability applicable to various 
inverse problems by what is assigned to a variable and the algorithm is realized by partial count, it also 
has the advantage that parallel processing hard instrumentation of light etc. is easy, and can perform 
high-speed processing. 
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[0062] Therefore, at step 102 of drawing 3 , the minimization is performed with an energy function, and 
when discontinuity detection does, the profile part (discontinuity) of an image is extracted by the neural 
network who used the above-mentioned Rhine process. Namely, according to (2) types, if an internal- 
state variable is updated, the minimum value will be calculated by the gross energy of (1) type always 
decreasing, hij and vij which were obtained at the time of this min are equivalent to the discontinuity 
(profile) of horizontal and a perpendicular direction. 

[0063] Next, at step 104, a color field setup of the similar color in a subject-copy image is made. This 
similar color setting processing is processing which carries out labeling of for example, an affiliated 
color, the same color and the color specified in the combination of a color defined beforehand, or the 
color group on a subject-copy image. For example, there is a method of performing integrated 
processing etc. by the repetitive mold field dilation based on the distance found about each pixel on a 
subject-copy image in the color space (henceforth a HLS color space) of the system of coordinates 
which consist of the hue value shaft, saturation value shaft, and lightness value shaft which intersect 
perpendicularly respectively. 

[0064] Specifically, any one pixel is first chosen from on a subject-copy image. Next, 1 pixel around the 
pixel (about [ so-called ] 8 pixel [ 1 ]) is chosen, and the label is given when the selected pixel is 
contained to the field to which which label was already given. On the other hand, when not contained, 
the distance in the HLS color space of two selected pixels, i.e., the lightness of two pixels, saturation, 
and the similarity of a hue are searched for. If distance excels, similarity is low, and similarity will be 
high if distance is short. The same label is given to two pixels noting that two pixels are similar, when 
distance is under a predetermined value (labeling). At the time beyond a predetermined value, the 
above-mentioned processing is performed about other about eight pixels, without carrying out labeling 
noting that it is not similar. After all about eight pixels are completed, 1 pixel of the outermost periphery 
of the field (it can set in a subject-copy image) where the same label was given is chosen, and the above- 
mentioned processing is repeated and performed. The above-mentioned processing is performed about 
all pixels. By carrying out sequential execution of the same label grant about the pixel of the perimeter 
where similarity is high from the pixel of the outermost periphery of the field where the same label was 
given, the color field by the pixel to which the hue on a subject-copy image, lightness, and saturation 
were similar is set up. In addition, the pixel group to which the same label was given calculates each 
average value of a hue value, a saturation value, and a lightness value, and makes the calculated average 
value the hue value, saturation value, and lightness value of a label. 

[0065] In addition, since the color field specified with the same label does not contain a discontinuous 
part, it is not divided, the discontinuity, i.e., the profile, detected at step 102 at least. 
[0066] As mentioned above, at step 104, after a color field is set up with the similarity of a hue, 
lightness, and saturation about the pixel which the subject-copy image adjoined, in the following step 
106, the class division of the subject-copy image is carried out. This class division is processing which 
defines relation between the color fields of the similar color set as the above-mentioned subject-copy 
image. That is, although the pixel group by the same or similar hue which is the color field where one 
label was given, lightness, and saturation has high possibility of being the same body, the color field of 
the similar color which is in the location distant on the subject-copy image, without adjoining has high 
possibility of being the body of the same class. Then, the value and range of the hue for carrying out the 
class division (class division) of the similar color field beforehand, lightness, and saturation are 
appointed, and the class division of the subject-copy image is carried out by classifying a color field in 
the value and range which were appointed. 

[0067] In addition, although the gestalt of this operation explains the example which defines and carries 
out the class division of the relation between the color fields of the similar color set as the subject-copy 
image, since discontinuity information is equivalent to the profile part of a body image, the class 
division of it can also be carried out by the similarity of discontinuity information (profile). 
[0068] If the subject-copy image 40 shown in drawing 4 is specifically made into an example, a class 
division can be carried out at the person class CC including the environmental class CB containing the 
scenery class CA containing a crest 42, Oyama 44, and clouds 46, the scattered flowers 50, and a garden 
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tree 48, and a person 52. 

[0069] At the following step 108, an image is extracted for every class division. If the subject-copy 
image 40 shown in drawing 4 is specifically made into an example, the scenery class CA will be 
extracted as a crest 42, Oyama 44, and a class image 54 containing clouds 46. This class image 54 
maintains the physical relationship on a crest 42, Oyama 44, and the subject-copy image of clouds 46 as 
it is, and a larger field is set up so that these images may not be divided at least. In addition, the 
background on the subject-copy image 40 may be made to remain as it is in the class image 54, and the 
above-mentioned image may be stuck on a new background, for example, the background color defined 
beforehand. Similarly, the class image 56 containing the flower 50 and garden tree 48 to which the 
environmental classes CB were scattered, and the person class CC are extracted as a class image 58 
including a person 52. 

[0070] Two or more body images may be contained on the class image extracted as mentioned above. 
Then, although a detailed example is mentioned later, at step 1 10, one class image is first specified 
among the class images extracted at the above-mentioned step 108. Next, in step 1 12, one body image is 
extracted and discontinuity information is outline-ized using a Bezier curve at the following step 1 14. 
Termination of outline-izing asks for whenever [ to the background of a body image / penetration ] in 
the following step 115 using discontinuity information. 

[0071] At the following step 1 16, the color by which the color information on whenever [ profile / which 
was outline-ized / and penetration ], and a body image (texture information) was encoded describes a 
body image. At the following step 1 1 8, it judges whether all the body images contained in the class 
specified at the above-mentioned step 110 were extracted, and when it is denied when a non-extracted 
body image remains, and the extract of return and all body images is completed to step 1 12, it 
progresses to the following step 120. At step 120, it judges whether all the above-mentioned processings 
of the class image contained in the subject-copy image were ended, and when it is denied when an 
unsettled class image remains, and processing of return and all class images is completed to step 1 10, 
this routine is ended. 

[0072] In addition, when texture information is beforehand memorized as information including the 
color information on a body image, processing of the above-mentioned class division etc. is 
unnecessary, and should just use texture information as it is. 

[0073] Next, the processing after step 1 10 of drawing 3 is concretely explained with reference to 
drawing 4 . A crest 42, Oyama 44, and the body image of clouds 46 are contained in the class image 54 
of the scenery class CA shown in drawing 4 . These body images are separated and extracted from the 
class image 54. This separation extract is made using discontinuity information. Since discontinuity 
information is equivalent to the profile part of a body image, it sets up the field surrounded by 
discontinuity information as a body image. Although the field surrounded by this discontinuity 
information is the closed region and closed space by the curve and curved surface which connected the 
break point equivalent to the profile part of a body image, it is limited to neither one closed region nor 
closed space. That is, you may be the group which consists of two or more closed regions and closed 
space. In addition, the closed region and closed space by the curve and curved surface which connected 
the break point may generate a closed region and a closed space according to the curve and curved 
surface which were connected and acquired on the parametric curve or the curved surface so that it may 
mention later. 

[0074] For example, three fields surrounded by discontinuity information are set up in the class image 
54. A sequential setup of these three fields is carried out, each value of a hue, lightness, and saturation is 
extracted about discontinuity information and the color information on the field surrounded for 
discontinuity information, i.e., the pixel in a field, and the crest image 60, the Oyama image 62, and the 
cloud image 64 are extracted. In addition, in the example of drawing 4 , what added the background 
image to each extracted image about each of the crest image 60, the Oyama image 62, and the cloud 
image 64 is set up as a body image. 

[0075] Similarly, the flower image 66 and the garden tree image 68 are extracted from the 
environmental class CB, and a person's head image 70, arm image 72, idiosoma image 74, and leg image 
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76 are extracted from the person class CC. 

[0076] Next, it is the following, and the profile (discontinuity information) of the extracted image is 
made and outline-ized. In order to outline-ize discontinuity information, description adopts an easy 
parametric curve and adopts a Bezier curve with simple treatment among parametric curves with the 
gestalt of this operation especially. Therefore, the profile (discontinuity information: hij, vij) of the 
extracted image is made binary, and it outline-izes using a Bezier curve. The formula which defines this 
Bezier curve P (t) is shown in the following (6) types. 
[0077] 
[Equation 4] 

p(t)-$,W(t) ... ( 6 ) 

(-0 

[0078] However, Bin(t) = (1-t) and Bin-l(t)+t-Bi-ln-l (t) 
Bjn(t) = 0G**n) 
= 1 G=n) 

Pi: Control point (break point) 

t : parameter [0079] Thus, even if it expands a body image or reduces by outline-izing the profile part of 

a body image, the profile configuration will be maintained. And it can describe in the format which can 

reuse a body image by describing a body image by this outline-ized profile and the color using the color 

information on a body image (encoding being desirable) (components-izing). 

[0080] Next, it asks for whenever [ to the background of a body image / penetration ] using the 

discontinuity information equivalent to the profile of the extracted image. With the gestalt of this 

operation, it asks [ whenever / Mh (ij) and vertical penetration ] for each of Mv (ij) using the following 

(7) types whenever [ horizontal penetration ]. 

[0081]Mh(ij)=beta(mij) 

Mv (ij) =beta (nij) ... (7) 

However, beta(): Even if it expands a body image or reduces by asking for whenever [ penetration / of 
nonlinear functions, such as a sigmoid function, or linear-function 0 <=Mh(ij) <=10 <=Mv(ij) <=1, thus 
a body image ], it becomes possible to make the profile part of a body image melt into a background. 
And a body image can be described in a reusable format [ be / no sense of incongruity ] by describing a 
body image by this outline-ized profile, whenever [ penetration ], and the color using the color 
information on a body image (encoding being desirable) (components-izing). 
[0082] As a body image which outline-ized the profile part of a body image and asked for whenever 
[ penetration ], the crest image 60 shown in drawing 10 (A) is made into an example, and whenever 
[ penetration / of a body image ] is explained further. Ridgeline 60A of a crest expresses the profile part, 
and has whenever [ penetration ]. Whenever [ penetration ] changes from maximum gradually between 
the minimum values so that I may be understood from the above-mentioned formula. Drawing 10 (B) is 
the enlarged drawing of Field Ar, and shows transition of concentration change of ridgeline 60 A with a 
contour line. Drawing 10 (C) shows the concentration change in the i-i cross section in Field Ar (refer to 
drawing 10 (B)). If the field Ar containing ridgeline 60A of a crest is referred to in the crest image 60 
shown in drawing 10 (A) as shown in drawing 1010 (B), concentration will become small gradually so 
that it may melt into background 60B, as near ridgeline 60A which is the boundary of an original crest 
image separates from ridgeline 60A (refer to drawing 10 (C)). Therefore, it will melt into a background 
gradually, without the profile part of a body image serving as a clear ridgeline. 
[0083] In addition, although the example which described the body image above by the outline-ized 
profile and the color using the color information on a body image was explained, this invention is not 
limited to this and may give information when describing a body image. For example, when images, 
such as a line drawing and a pattern, are contained in the body image or fixed processing of mask 
processing, focal processing, etc. is performed to the body image, the image and processing may be 
made to belong. That is, it can subdivide by patterns, such as a line and gradation, etc., or the description 
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about a body image can also be subdivided by the performed processing. 

[0084] Moreover, although the case where a parametric curve was used was explained above, it is not 
limited to a parametric curve and other approximation curves may be used. 

[0085] Next, the case where the components-ized body image is expanded is explained. The case where 
the color information on a body image is encoded using a NURBS (Non Unifom Rational B-Spline) 
curved surface is made into an example, and the following explanation explains. The formula which 
defines NURBS as this parametric curved surface is shown in the following (8) types. 
[0086] 
[Equation 5] 

S (u, v) = i^f2_ • ■ . (8) 

w ,^;( M )iv/(v) 



i-O i-0 



fflU 

*,'('>! 0, <*,♦.) 
-0 (f^f i+1 ) 



[0087] Hereafter, the case where resolution is expanded twice (4 times as many surface ratio as this) 
using 3rd NURBS with reference to drawing 8 and drawing 9 is explained. 

[0088] At step 200 of drawing 9 , reading of the subject-copy image equivalent to step 100 of drawing 3 

is made. Outline-ization by which the discontinuity detection by the Rhine process by which this read 

subject-copy image is equivalent to step 102 of drawing 3 R> 3 in the following step 202 is made, and is 

equivalent to step 1 14 of drawing 3 in the following step 204 is made. At the following step 205, the 

operation of whenever [ equivalent to step 1 15 of drawing 3 / penetration ] is made. 

[0089] Next, at step 206, the outline-ized discontinuity information is expanded and whenever 

[ penetration ] is expanded in the following step 207 (it mentions later for details). At the following step 

208, resolution is enlarged using the discontinuity information and the parametric curved surface which 

were expanded. And the image with which resolution was enlarged is outputted in the following step 

210. 

[0090] Drawing 8 shows the configuration which uses each pixel as a control point Pij (0<=i<=3, 
0<=j<=3) as what constitutes an image 80 from a pixel of four every direction (a total of 16 pieces) 
which constitutes some subject-copy images. A curve 82 is a curve obtained when the discontinuity 
information extracted according to the Rhine process which gave [ above-mentioned ] explanation is 
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expanded as it was, and a curve 84 is a curve obtained when it expands, after outline-izing discontinuity 
information which gave [ above-mentioned ] explanation. 

[0091] As each pixel value, white and control points P30, P21, P31, P12, P22, P32, P03, P13, and P23 
are set up by the white pixel as that whose others the one half is black pixels, and, as for an image, a 
black value is set up for control points POO, P10, P20, P01, PI 1, and P20. 

[0092] With the gestalt of this operation, a pixel is newly added as an ideal point S in a control point 
POO - P33 at the time of the above-mentioned expansion. For example, an ideal point Smn (0<=m<=3, 
0<=n<=3) is added within limits surrounded at control points PI 1, PI 2, P21, and P22. 
[0093] The pixel value of the these-added ideal point S is calculated with reference to the pixel value 
(pixel value of a control point) of the subject-copy image of the perimeter. With the gestalt of this 
operation, a pixel value is determined in consideration of the discontinuity in a subject-copy image. That 
is, the pixel value of the pixel beyond the curve 84 which is discontinuity information shall not be used. 
For example, when you calculate the pixel value of an ideal point S20, since the straight line which 
connects an ideal point S20 does not intersect a curve 84, it refers to, but since the straight line which 
connects an ideal point S20 intersects a curve 84, don't refer to the control point P22 for a control point 
PI 1 . Count is made to reflect the weight in (8) types showing Above NURBS in a detail by setting it as 
"0" more. By this, the pixel value of an ideal point S20 turns into a value of a white pixel. Other ideal 
points are calculated similarly and a pixel value is calculated. 

[0094] Thus, if discontinuity information is outline-ized and is expanded, the edge by which dotage was 
controlled can be obtained. By this, the jaggy (the so-called notch) of the edge at the time of image 
expansion can be stopped. 

[0095] Next, it asks for whenever [ penetration ] using discontinuity information. Whenever 

[ penetration ] exists only in the discontinuity information which exists between the pixels before 

expansion. For this reason, the value (discontinuity information) over whenever [ whenever / 

penetration / between the pixels after expansion ], i.e., discontinuity, is calculated first. 

[0096] Whenever [ before expansion / penetration ] is Mv02 whenever [ penetration / of the 

perpendicular direction using the discontinuity information which exists / whenever / penetration / using 

the discontinuity information which exists among the control points P20 and P21 corresponding to a 

curve 82 / whenever / penetration / using the discontinuity information which exists between Mh20 and 

control points PI 1 and P12 / between Mvl 1 and control points P02 and P12 whenever / penetration / 

using the discontinuity information which exists between MM 1 and control points PI 1 and P21 ]. 

[0097] Next, it asks for whenever [ corresponding to the curve 84 after expansion / penetration ]. 

Weighting of whenever [ this penetration ] is carried out with the distance between whenever 

[ penetration / which exists in contiguity ], and it asks for whenever [ new penetration ]. It can ask for 

whenever [ new penetration ] by one [ following (9) and / at least ] formula of (10) types. 

[0098] 

Mvi-k=alpha-Mvi+beta-Mhj ... (9) 
Mhi-k=gamma-Mvi+epsilon-Mhj ... (10) 

however i : horizontally the location of whenever [ before the latest expansion / penetration ] to a criteria 
side The multipliers beta and epsilon which express the proportionality of distance with perpendicularly 
horizontal the value alpha and gamma: drawing 8 showing the location of whenever [ after the value 
k:expansion showing the location of whenever / penetration / before the latest expansion / penetration ] 
to the value j criteria side to express : on the multiplier concrete target showing the proportionality of a 
horizontal distance of drawing 8 Whenever [ penetration / of the ideal point S20 circumference ], a 
degree can have Mv 11-1, Mv 1 1-2, and Mh 20-1, and they can be calculated. 
[0099] 

Mvl 1-1=0.4, Mvl 1+0.6, and Mh20Mvl 1-2=1.0, Mvl 1+0.0, and Mh20Mh 20-1 - Mv 11-1 considers 
the location Pp between an ideal point S20 and a control point P21 as criteria of distance, and asks 
[ =0.25, Mvl 1+0.75 and whenever / Mh20 penetration / whenever / penetration ] for the distance ratio to 
each of Mh20 whenever [ Mvl 1 and penetration ]. When spacing of a control point is assumed to be "1", 
the distance ratio to Mvl 1 is 1/3 whenever [ penetration ], and the distance ratio to Mh20 is 1/2 
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whenever [ penetration ]. It will be set to alpha= 0.4 and beta= 0.6, if it standardizes so that these sums 
may be set to "1." Whenever [ other penetration ] can be found similarly. 

[0100] Here, in not using whenever [ penetration ], when calculating the pixel value of an ideal point 
S20, since the straight line which connects an ideal point S20 does not intersect a curve 84, it refers to a 
control point PI 1, but since the straight line which connects an ideal point S20 intersects a curve 84, a 
control point P22 is not referred to. When setting the weight wij in (8) types which express NURBS 
when referring to other control points as "1" and referring to it in a detail, weight wij is made to reflect 
in count by setting it as "0." By this, the pixel value of an ideal point S20 turns into a value of a white 
pixel. 

[0101] On the other hand, whenever [ penetration ] is used with the gestalt of this operation. For this 
reason, (following 1 1) and following (12) types are used for the weight wij in (8) types. However, 
whenever [ over the control point which does not intersect discontinuity information / penetration ] is 
wij=l like the above. 
[0102] wij=l-Mvi-k...(ll) 
wij=l-Mhi-k...(12) 

In addition, in the case of an ideal point S20, count of weight wij uses whenever [ discontinuity 
information / which the straight line which connects between each control point intersects / i.e., 
penetration corresponding to part for intersection of curve 84 after expansion, ]. Moreover, in 
intersecting two or more discontinuity information, it shall use the maximum of whenever 
[ penetration ]. 

[0103] Thus, a body image becomes expandable [ in consideration of whenever / penetration / with a 
background ] by calculating by carrying out weighting to (8) types. 

[0104] In addition, above, although the example of a white pixel and a black pixel was explained, when 
it is the pixel which has concentration, and the pixel which has a color, a pixel value may be calculated 
by calculating the forecast by equalization or inclination. 

[0105] Thus, the profile of the body image which detects discontinuity information using the Rhine 
process and is contained in a subject-copy image using the discontinuity information in a subject-copy 
image is extracted. Moreover, an image can be generated, without the profile of a body image fading or 
becoming coarse, even if it is the case where outline-ized the profile of a body image using discontinuity 
information, and a resolution change, for example, expansion, is made. Furthermore, even if it is the 
case where carried out enlarging or contracting, compounded with other images, or it compounds for a 
background with resolution modification, for example, expansion, an image is generable, since it is 
asking for whenever [ penetration / of a body image ] using discontinuity information so that a body 
image may melt. By this, a body image can be components-ized, and can be described, without being 
dependent on resolution, and reuse can be made easy. 

[0106] In addition, although explanation of taking into consideration about the component of a pixel 
value was omitted with the gestalt of the above-mentioned implementation, to each component of RGB, 
the above-mentioned processing may be performed or you may carry out to the component of a hue or 
saturation, for example. 

[0107] Moreover, although the case where enlarging or contracting was performed in the same image 
was explained, also when starting the object which removed a part for the background in a body image 
and compounding with other background images, it can apply. In this case, what is necessary is to 
memorize whenever [ penetration / when starting ], and just to use whenever [ penetration / which was 
memorized ], when compounding with other background images. By doing in this way, it becomes 
generable [ a synthetic image with more little sense of incongruity ]. 

[0108] Next, transform processing which generates dynamic-image data is explained using coding and 
enlarging or contracting of above-mentioned image data. In addition, the following manipulation 
routines are stored in record media, such as FD, are offered as application which can be performed by 
computer, and may be made to perform with activation directions. 

[0109] First, if a power source is supplied to image transformation equipment 10, the manipulation 
routine shown in drawing 1 will be performed. In addition, this manipulation routine is stored in record 



http ://www4 . ipdl .ncipi . go.j p/cgi-bin/tran_web_cgi_ej j e 



12/20/05 



JP 5 2000-201358,A [DETAILED DESCRIPTION] 



Page 15 of 16 



media, such as FD, is offered as application which can be performed by computer, and may be made to 
perform with activation directions. 

[0110] At step 300 of drawing 1 , two or more target static images are read as a dynamic image. At the 
following step 302, a criteria image is read among two or more read static images, and a criteria image is 
encoded in the following step 304. The first static image can be used for this criteria image in time in 
two or more static images. Moreover, processing of step 304 can be performed by the coding processing 
(refer to drawing 3 ) mentioned above. 

[0111] Termination of coding of a criteria image reads the static image which follows a criteria image 
among two or more static images in the following step 306. It asks for the difference of a criteria image 
and the static image read at step 306 at the following step 308. This step 308 is because it specifies how 
much the read static image changed from the criteria image. 

[0112] next, the difference for which it asked the account of a top in step 3 10 — the body image (change 
object) which changed from the criteria image using the value is specified, and the vector information on 
a body image is extracted in the following step 312. When the read static image changes from a criteria 
image, it appears as difference, the difference ~ the similarity of the body image contained in a criteria 
image is searched for from a value, a location, and a configuration, and it asks for the location 
fluctuation and magnitude fluctuation. Let these be vector information. In addition, when vector 
information is memorized beforehand, processing of this step is unnecessary. 

[0113] At the following step 3 14, a change object is encoded about the static image read at step 306. It is 
good only for the information and vector information that it means carrying out this coding based on a 
criteria image. That is, it is because the read static image is reproducible if there are a criteria image and 
vector information. 

[0114] Each processing is altogether performed about two or more target static images as repeat) and a 
dynamic image until it carries out affirmative judgment of the processing of the above-mentioned step 
306 to the step 314 at the (step 316. In addition, in the above-mentioned processing, although vector 
information was searched for from the criteria image, an adjacent static image may be followed. In this 
case, the body image contained in the last static image becomes what was changed with vector 
information from the body image contained in a criteria image. Moreover, the body image which was 
not contained in the criteria image but appeared newly in two or more static images can be newly 
encoded as a body image at the time (when difference without the above-mentioned similarity is called 
for). 

[0115] After the above-mentioned processing is completed about all static images, in the following step 
3 1 8, dynamic-image data are generated by compounding the data of each static image encoded above. 
This composition may arrange data in order, and may generate dynamic-image data, and dynamic-image 
data may be generated by carrying out a data description according to the format defined beforehand. 
[0116] Next, processing of drawing 1 is concretely explained with reference to drawing 1 1 , drawing 
12 , and drawing 13 R> 3. An example which constituted the dynamic image as a subject-copy image 
with two or more static images 40A, 40B, and 40C was shown in drawing 1 1 (A) - drawing 1 1 (C). In 
this dynamic image, although a crest 42 and the body image of flower 50 grade are changeless, the 
person 52 is approaching in the crest 42 palmette flower 50 direction gradually. Magnitude is changed 
while, as for a person's 52 body image, changing a location at this time. 

[0117] therefore, a person's 52 body image shown in image 40S and drawing 12 (B) of the background 
excluding the person 52 from the subject-copy image 40 (static-image 40A) which shows dynamic- 
image data to drawing 12 (A) and its vector information 53 - since - it can express. Vector information 
expresses the information about a location and magnitude. 

[01 18] By searching for discontinuity information about the subject-copy image 40 (static-image 40 A), 
the profile and the description part of a body image can be extracted. A person's 52 up enlarged drawing 
was shown in drawing 13 (A). If this discontinuity information is searched for, as shown in drawing 13 
R> 3 (B), the description of a person's profile and expression, or each part can be extracted. That is, 
although change or the description of expression cannot be found out only with a person's profile, it can 
ask for the description part of the image by searching for the discontinuity information inside an image. 
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[0119] Drawing 14 shows the condition (all busy condition) that a person's (only a head object) interior 
of an image changed. Drawing 14 (A) searches for discontinuity information about an image when 
opening opening, and drawing 14 (B) searches for discontinuity information about an image when 
having closed opening. The description has appeared on the outskirts 55 of opening so that I may be 
understood from drawing. 

[0120] Thus, the description part (discontinuity information) of the extracted image is outline-ized. Even 
if it expands a subject-copy image and a body image or reduces by outline-izing the profile part of a 
body image, and the discontinuous part of the interior, the profile configuration and description part will 
be maintained. For this reason, even if it is a time of compounding color information (texture 
information) into a profile configuration or the description part, it is not influenced by resolution and an 
image can be expressed. 

[0121] As explained above, with the gestalt of this operation, the profile of the body image which 
detects discontinuity information using the Rhine process and is contained in a subject-copy image using 
the discontinuity information in a subject-copy image, and the discontinuity inside an image are 
extracted. An image can be generated without the profile of a body image fading or becoming coarse, 
even if it is the case where outline-ized the profile of a body image, and discontinuity inside an image 
using discontinuity information, and a resolution change, for example, expansion, is made. 
[0122] Moreover, since the image data showing a dynamic image can describe the profile of an image, 
and the discontinuity inside an image using the outline-ized data and texture information, such as color 
information by which endocyst is carried out to a profile, even if it is the case where transmit image 
data, are a transmitted side and an image is reconstructed, it can offer the quality image for which an 
image does not fade, or does not become coarse and it does not depend on resolution. Moreover, even if 
it is the case where enlarging or contracting of the image is carried out, the quality image independent of 
resolution can be offered. 
[0123] 

[Effect of the Invention] As explained above, while outline information describes the profile of a body 
image according to this invention Since images including the texture information which expresses a 
body image according to a profile are encoded When reproducing two or more subject-copy images or 
body images as a dynamic image, even if it is the case where it expands or reduces The profile of the 
original body image can be maintained without being dependent on resolution, and it is effective in 
conversion of the image data which does not produce sense of incongruity in dynamic-image playback 
being possible. 

[0124] moreover, since images including the texture information which expresses a body image 
according to a profile are encoded while outline information describes the profile of a body image, and 
the discontinuity inside a body image, there is effectiveness of the ability to also make the detailed 
actuation inside the body image of expression change of a person reproduce faithfully from fluctuation 
of discontinuity information. 
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* NOTICES * 
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1 .This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is the flow chart which shows the flow of the processing performed with the image 
transformation equipment concerning the gestalt of operation of this invention. 
[Drawing 2] It is the block diagram showing the outline configuration of the image transformation 
equipment concerning the gestalt of operation of this invention. 

[Drawing 3] It is the flow chart which shows the flow of coding processing with the image 
transformation equipment concerning the gestalt of operation of this invention. 

[Drawing 4] It is the image Fig. showing the process in which a body image is extracted from a subject- 
copy image. 

[Drawing 5] It is an explanatory view for explaining the Rhine process. 

[Drawing 6] It is a conceptual block diagram for explaining the neural network using the Rhine process. 
[Drawing 7] It is an explanatory view for explaining the Rhine process applied to the image. 
[Drawing 8] It is an explanatory view for explaining the configuration of the image when expanding 
resolution using NURBS. 

[Drawing 9] It is the flow chart which shows the flow of the processing to which a subject-copy image is 
expanded. 

[Drawing 10] It is an explanatory view for explaining whenever [ penetration ] using a crest image. 
[Drawing 11] It is the image Fig. which disassembled the dynamic image into the static image. 
[Drawing 12] It is an explanatory view for explaining vector information. 

[Drawing 13] It is an image Fig. showing the body image for explaining discontinuity information. 
[Drawing 14] It is an explanatory view for explaining the discontinuity information inside a body image. 

[Description of Notations] 

10 Image Transformation Equipment 

30 Image Reader 

40 Subject-Copy Image 

54 Class Image 

60 Crest Image (Body Image) 
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precisely. 
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3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1] The dynamic-image viewing window which is the dynamic-image method of presentation in 
the equipment which displays the structure of a dynamic image and a dynamic image, continues in order 
of a frame and displays the inputted dynamic image, The panorama image display window which 
displays the panorama image of this dynamic image serially, The dynamic-image method of presentation 
characterized by displaying on a display the icon image display window which indicates this panorama 
image by list as a break and a still picture by both every changing point of a dynamic image, and both 
[ either or ] for every assignment unit. 

[Claim 2] A dynamic-image viewing window according to claim 1 is the dynamic-image method of 

presentation characterized by displaying the dynamic image inputted from image pick-up equipment, the 

dynamic-image regenerative apparatus, or the external information storage device. 

[Claim 3] A panorama image display window according to claim 1 is the dynamic-image method of 

presentation characterized by carrying out a scaling and making it display that a panorama image is 

arranged at the core of the above-mentioned panorama image display window, and the above-mentioned 

whole panorama image is settled in the above-mentioned panorama image display window. 

[Claim 4] The changing point of a dynamic image according to claim 1 is the dynamic-image method of 

presentation characterized by being both a point changing [ cut ], or dissolve both [ either or ]. 

[Claim 5] A still picture according to claim 1 is the dynamic-image method of presentation characterized 

by carrying out a scaling to predetermined size. 

[Claim 6] An icon image display window according to claim 1 is the dynamic-image method of 
presentation characterized by doubling the information about the start time or end time of a dynamic 
image represented by the above-mentioned still picture, and making it display while indicating the still 
picture by the list. 

[Claim 7] An icon image display window according to claim 1 is the dynamic-image method of 
presentation characterized by also making the information which shows the moving trucking of camera 
actuation superimpose, and making it display on the above-mentioned still picture while indicating the 
above-mentioned still picture by the list. 

[Claim 8] An icon image display window according to claim 1 is the dynamic-image method of 
presentation characterized by displaying the information which shows the moving trucking and transit 
time of a view by camera actuation about each of the above-mentioned still picture while indicating the 
above-mentioned still picture by the list. 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the dynamic-image method of presentation, generates a 
suitable representation screen from the dynamic-image group of photoed a large number in detail, and 
relates to the approach of showing a user the chart of a still picture. 
[0002] 

[Description of the Prior Art] In recent years, improvement in the speed and large-capacity-izing of a 
calculating machine are made into a background, and the database system and the presentation tool 
which can treat dynamic-image information which was not able to be treated, such as a movie and video, 
attract attention conventionally. However, only a required part is looked for out of the dynamic-image 
information which has huge amount of information, or a time-consuming activity processes it for a user. 
Then, it is for example, the Information Processing Society of Japan paper magazine as the activity 
derating approach by the computer. Vol.33 and No.4, There is an approach shown by "the automatic 
indexing method and body heuristics" in a color video image, JP,4-1 1 1 181, A "the change check attitude 
method of a dynamic image", etc. 

[0003] This is dividing a dynamic image automatically per cut, creating the list of an image representing 
each cut, and using it like the index of books, and is the approach of making it into an aid of retrieval of 
a dynamic image or edit. According to this, the contents of the dynamic image are on a list, and since it 
is easy to find a desired scene since it can grasp at a glance, and it can treat in the good unit of a cut 
which can be divided, rough edit can be performed simply. 
[0004] 

[Problem(s) to be Solved by the Invention] In such an approach, it is necessary to choose one frame 
considered to represent the cut most to one cut. It fixes, and if it is a dynamic image from the photoed 
video camera, it will be satisfactory by usually using the head frame of a cut as a representation screen. 
However, if panning which shakes a video camera at right and left, and camera actuation of chill 
TINGU shaken up and down are performed, since scenery various in one cut is included, it becomes 
very difficult to choose the optimal representation screen of one sheet from them. 
[0005] By the conventional approach, it is only choosing the head screen of a cut and the information in 
the middle of a photograph having been taken when camera actuation was performed was lost. 
Therefore, there was a case where a scene important at the time of retrieval of a dynamic image or edit 
was overlooked. 

[0006] The purpose of this invention has a representation image in offering the dynamic-image method 
of presentation which indicates by list, without also making missing the intermediate information at the 
time of camera actuation. 
[0007] 

[Means for Solving the Problem] In order to solve the above-mentioned technical problem, a dynamic 
image is displayed by the dynamic-image viewing window which continues in order of a frame and 
displays the inputted dynamic image, the panorama image display window which displays the panorama 
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image of this dynamic image serially, and the icon image display window which indicates this panorama 
image by list as a break and a still picture by both every changing point of a dynamic image, and both 
[ either or ] for every assignment unit. 

[0008] The time whereabouts of a dynamic image is clarified by doubling and displaying a hour entry 
with an icon image especially. Moreover, it is shown clearly with what kind of view camera actuation 
was performed, and by showing the information about camera actuation also shows clearly the gaze time 
amount in a certain view, and the rate of camera actuation. 
[0009] 

[Embodiment of the Invention] Drawing 1 is the block diagram of the system configuration for realizing 
this invention. The display units 1, such as CRT, display the output screen of a computer 4. The 
instruction to a computer 4 can be performed using the input units 5, such as a keyboard and a pointing 
device. The dynamic-image regenerative apparatus 10 are an optical disk, a videocassette recorder, etc. 
Serially, the video signal outputted from the dynamic-image regenerative apparatus 10 is changed into 
digital image data by A/D converter 3, and is sent to a computer 4 by it. Digital image data goes into 
memory 9 through an interface 8, and is processed by CPU7 according to the program stored in memory 
9. When the number (frame number) is assigned to each frame of a dynamic image sequentially from the 
head of a dynamic image, the corresponding dynamic image of a scene is reproduced by sending a frame 
number to the dynamic-image regenerative apparatus 10 with the control line 2. According to the need 
for processing, various information can be accumulated in the external information storage device 6. As 
information recorded on the external information storage device 6, the index data 6-2, such as a video 
data 6-1 and an icon image, the panorama image data 6-3, and the dynamic-image display program 6-4 
are contained. The dynamic-image display program 6-4 is read to memory 9, and is performed. 
[0010] In the system configuration of the above dynamic-image display, it inputs one dynamic image 
from the dynamic-image regenerative apparatus 10 at a time into memory 9 by the dynamic-image 
display program 6-4. The inputted dynamic image is displayed on a display 1 . Furthermore the camera 
actuation in a dynamic image is extracted, a panorama image is created, and a panorama image is also 
displayed on a display 1 with a dynamic image. Here, a panorama image means the image which 
connected superficially and created the continuous frame according to the mutual amount of location 
gaps. And when a cut change of the inputted dynamic image is detected, the scaling of the panorama 
image in preparation is carried out until now so that it may become fixed size, an icon image is created, 
and on a display 1, it is made a list and displays. The dynamic image obtained while processing above, 
an icon image, and a panorama image are stored in the external information storage device 6 if needed. 
[001 1] In addition, with the gestalt of this operation, although the dynamic image is inputted from 
dynamic-image regenerative apparatus, such as VTR and DVD, they may be the dynamic image of the 
television broadcasting using a tuner, a dynamic image from a video camera, and a dynamic image 
further stored in external information storage devices, such as a hard disk. 

[0012] Drawing 2 is the example of the display screen of the dynamic image on the display 1 shown in 
drawing 1 . A window 20 is displayed by the dynamic-image display program 6-4. The windows 21 in a 
window 20 are [ a panorama image display window and the window 23 of a dynamic-image viewing 
window and a window 22 ] icon image display windows. The check carbon button 24 controls the 
display mode of the icon image display window 23. The check carbon button 25 controls the processing 
mode of whether to classify a dynamic image in respect of a cut change. The carbon button 26 is a 
toggle switch and controls initiation and termination of processing of a cine mode display. In case a 
carbon button 27 outputs a panorama image current in preparation to the external information storage 
device 6, the depression of it is carried out. 

[0013] The panorama image 22-1 of a window 22 carries out serial generation at every input of a 
dynamic image, and displays. In that case, it positions in the center of a window, and a scaling is carried 
out and it displays so that a window 22 may not be overflowed. Thereby, the panorama image creation 
situation of the dynamic image under current input can understand a user correctly. 
[0014] The point or user of a cut change expresses the icon image 23-1 of a window 23 as the timing 
which took out the signal of termination with a carbon button 26. The scaling of this icon image 23-1 is 
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carried out so that it may become fixed size about a panorama image in preparation. The start time of 
panorama image creation is displayed on the icon image bottom. This may be a time code which a 
dynamic-image regenerative apparatus outputs. Since these icon images are panorama images, i.e., the 
static image of a wide angle, generated using all the frame image information obtained as a result of 
camera actuation, a user does not overlook the information on a dynamic image. 

[0015] Another display gestalt of the icon image display window 23 when the check carbon button 24 of 
drawing 2 is pushed on drawing 3 is shown. That is, the moving trucking of the view by camera 
actuation is displayed as a graphic form 23-1-1 on the icon image 23-1. More detailed information 
[ say / whether camera actuation was performed on the left from that camera actuation was performed on 
the right from Hidari by this or the right ] can provide for a user, without reducing the display tooth 
spaces of an icon image. 

[0016] Drawing 4 is another example of a window 23 which offers information still more detailed than 
drawing 3 . That is, transit time etc. is unknown although the moving trucking of a view became clear in 
drawing 3 . In this method of presentation, the information about the transit time how much gaze time 
amount there were in which view location is also offered. It is [ the camera actuation information 23-2 ] 
attached, and it is displayed on an icon image. The camera actuation information 23-2 consists of a 
graph 23-2-1 of the moving trucking of a view, and a graph 23-2-2 of the transit time of a view. An axis 
of abscissa shows [ the migration length of the direction of Y, and the intersection of each shaft ] the 
migration length of the direction of X, and, as for the moving trucking graph 23-2-1, an axis of ordinate 
shows the zero at the time of panorama image generation initiation (black dot mark). On the other hand, 
a lower left corner is a zero at the time of panorama image generation initiation, an axis of abscissa 
shows time amount and, as for the transit time graph 23-2-2, an axis of ordinate shows the view total 
migration length. Therefore, the section when the transit time graph is level does not have view 
migration to the direction of time amount, and it means that camera actuation is fixed there. By 
observing such the section, a user can discover immediately the scene at which the photography person 
gazed intentionally. 

[0017] Drawing 5 is the flow chart of the dynamic-image display program 6-4 performed on the system 
shown by drawing 1 . 

[0018] Data 6-1-3 referred to by the program before explanation of the dynamic-image display program 
6-4 are explained. These data are shown in drawing 6 . n frame images are recorded following the 
header the dynamic-image data 6-1 described image size, compressed format, etc. to be. The index data 
6-2 contain the icon image data 6-2-1 and frame image complex data. The icon image data 6-2-1 
consists of attached information, such as a header which described the size and Number m of an icon 
image, and a time code attached to a m icon image and its icon image. The frame image complex data 6- 
2-2 consists of each inter-frame panorama image composition parameter (dx, dy) which adjoins the 
header which described the number of a synthetic parameter. A parameter (dx, dy) expresses the amount 
of location gaps with the frame image inputted before the frame image which carried out the current 
input, and one. The panorama image data 6-3 consists of the headers and panorama images which 
described the size of a panorama image. 

[0019] First, the dynamic-image display program 6-4 is read to memory 9, and CPU7 performs 
processing 60 first and makes an initial state the windows 21, 22, and 23 of drawing 2 . And the 
following processings are repeated and performed until closing is directed with a file menu 28. 
[0020] First, it is confirmed by processing 62 whether a carbon button 26 is in an initiation condition. If 
it is in an initiation condition, processing 63 will be performed, otherwise, processing 80 will be 
performed. 

[0021] In processing 63, the frame image which constitutes a dynamic image from a dynamic-image 
regenerative apparatus 10 is inputted, and it stores in memory 9. Next, the inputted frame image is 
displayed on a window 21 (processing 64). And the inputted frame image is stored as a video data 6-1 of 
the external information storage device 6 (processing 65). In processing 66, the condition of the check 
carbon button 25 judges whether it is cut division assignment. If it is in a "cut division assignment" 
condition, processing 67 will be performed and the point of a dynamic image changing [ cut ] will be 
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detected, in order to detect the point of a dynamic image changing [ cut ] — current and a color 
histogram with the frame before it — inter-frame correlation is searched for from difference, and there is 
the approach of considering the time of correlation being lost as a cut change. For example, artificers' 
JP,8-227462,A "the change check attitude method of a dynamic image and equipment" can be used. 
[0022] Processing 69 will be performed if the point changing [ cut ] is detected. In processing 69, the 
scaling of the panorama image 22-1 on display in a window 22 is carried out to fixed size, and it 
displays on a window 23. In processing 70, the condition of the check carbon button 24 judges whether 
it is path display assignment. In path display assignment, when it integrates with a panorama image 
composition parameter (dx, dy) to dx and it integrates with (dy) separately from the synthetic initiation 
frame of a panorama image to a termination frame, it displays in piles on an icon image in the form of 
the two-dimensional graph 23-1-1 of drawing 3 . The moving trucking of the view of a camera can be 
displayed by this. As another method of presentation, there is a graph 23-2 of drawing 4 . As for the 
moving trucking graph 23-2-1 of a view, a panorama image composition parameter (dx, dy) can 
generate the moving trucking graph 23-2-1 of a view by integrating with each of dx and dy by 
displaying the value which integrated with (dx, dy) separately from the synthetic initiation frame of a 
panorama image to the frame at the time on time series. The moving trucking graph and moving 
trucking graph of a view are adjoined and displayed on an icon image. In processing 72, an icon image 
is stored in the external information storage device 6, the panorama image in a window 22 is eliminated 
by processing 73, and it considers as an initial state. 

[0023] Processing 74 is processing which generates a panorama image. A panorama image can be 
attained by detecting the location gap (dx, dy) with the frame image inputted before the frame image 
which carried out the current input, and one, and compounding an image according to a location gap. 
This panorama image generation processing can apply artificers' Japanese Patent Application No. No. 
153303 [ nine to ] "a digital wide camera." The generated panorama image is displayed on a window 22 
(processing 75). At this time, a panorama image is arranged at the core of a window 22, and a scaling is 
carried out and it is displayed that the whole panorama image is settled in a window 22. The panorama 
image composition parameter (dx, dy) used for generation of a panorama image is stored in the external 
information storage device 6 (processing 76), and is used in the processing 71 explained previously. 
[0024] On the other hand, if a carbon button 26 is exit status (processing 62), it will judge whether the 
user pushed the carbon button 27 (see drawing 2 ) (processing 80). If pushed, the panorama image 22-1 
on display will be outputted to the external information storage device 6 as panorama image data 6-3 in 
a window 22. It enables this to acquire the panorama image at the arbitration time for which a user asks. 
[0025] Next, it is confirmed whether the pointing of the icon image currently displayed all over the 
window 23 was carried out with the pointing device 5 (processing 82). When pointing is carried out, 
only the frame section when the video data 6-1 already stored in the external information storage device 
6 corresponds is reproduced partially, sequential creation of the panorama image is carried out using the 
synthetic parameter 6-2-2 applicable to coincidence, and it displays on a window 21 in a window 22. A 
user can see easily the detailed panorama image and the corresponding dynamic image of the icon image 
which became small too much by the scaling by this. 

[0026] Although a dynamic image is classified for every point of a cut change and the icon image of a 
panorama image is used as the chart in the above example, respectively, there is a point called dissolve 
into a dynamic image that a scene changes gently in the dynamic image inputted from television 
broadcasting. Possibility of overwriting the frame image of another scene on such a panorama image 
created before it when it changed, the point was disregarded and the panorama image was created comes 
out. What is necessary is just to also make a dissolve point into the conditions of a partition of an 
animation in addition to the point of a cut change, in order to solve such a problem. In order to detect the 
point which changes gently, it can attain by investigating change of the image of several past. JP,9- 
65287,A "the description scene detection approach of a dynamic image and equipment" by artificers is 
applicable to detection of this dissolve point. 
[0027] 

[Effect of the Invention] Since the panorama image which classified the whole dynamic image at the 



http ://www4 . ipdl . ncipi . go j p/cgi-bin/tran_web_cgi_ej j e 



12/20/05 



JP,1 1-134 3 52 3 A [DETAILED DESCRIPTION] 



Page 5 of 5 



changing point, and generated it from the camera actuation information on the classified dynamic image 
is indicated by list according to this invention, it can display without leaving all the photographic 
subjects in the classified animation. In the list display of the conventional dynamic image, since one 
frame representing it was chosen and displayed about the classified dynamic image, there was a problem 
which overlooks the photographic subject included in frames other than a representation image, but in 
this invention, it is effective in such a problem being solved in order to generate a panorama image using 
the information on all frames. Furthermore, in order to superimpose on an icon image and to display the 
moving trucking of the view by camera actuation, Are effective in the ability of a user to understand to 
photography technique by seeing an icon image. In this invention Since animation display by the 
dynamic-image viewing window, the panorama image display under generation by the panorama image 
display window, and the still picture display by the icon image list window can carry out to coincidence, 
it is effective in an understanding of a user's dynamic image being promoted more. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is an example of the system configuration for realizing this invention. 

[Drawing 2] It is an example of the screen display of this invention. 

[Drawing 3] It is another example of the screen display of this invention. 

[Drawing 4] It is still more nearly another example of the screen display of this invention. 

[Drawing 5] It is an example of the flow chart of a dynamic-image display program. 

[Drawing 6] It is an example of the structure of the data stored in an external information storage device. 

[Description of Notations] 

1 [ - An input unit, 6 / - An external information storage device, 7 / -- CPU, 8 / - An interface, 9 / -- 
Memory, 10 / — A dynamic-image regenerative apparatus, 21 / - A dynamic-image viewing window, 
22 / — A panorama image display window, 23 / - Icon image display window. ] — A display, 3 - An 
A/D converter, 4 - A computer, 5 
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[Item(s) to be Amended] Claim 
[Method of Amendment] Modification 
[Proposed Amendment] 
[Claim(s)] 

[Claim 1] An input means to input the photoed dynamic image A display and the control section which 
processes an image It is the record medium which recorded the program which performs the image 
display approach equipped with the above. The step which sets the dynamic-image viewing window 
which indicates the frame which constitutes the dynamic image inputted into this equipment by 
sequential as this control section, and it is made to display on this display, The step which carries out 
sequential composition of the frame which constitutes this dynamic image, and creates a panorama 
image, The step which sets up the panorama image display window which displays this panorama image 
serially, and is displayed on this display, By either or both for every assignment unit into which this 
panorama image was inputted by every changing point and this equipment of this dynamic image, a 
break, The icon image display window which indicates the image of 1 or two or more break eyes of this 
break by list as 1 or two or more still pictures is set up, and it is characterized by having the step 
displayed on this display. 

[Claim 2] The record medium which recorded the program which performs the image display approach 
according to claim 1 characterized by providing the following The step which chooses the still picture of 
1 as the above-mentioned control section from Norikazu after being further indicated by list in the 
above-mentioned icon image display window in addition to the above-mentioned step, or two or more 
still pictures according to the directions into which it was inputted by the above-mentioned equipment 
The step which reproduces the frame section represented by the still picture of 1 with which the above- 
mentioned dynamic image was this chosen, and is displayed on the above-mentioned dynamic-image 
viewing window The step which creates the panorama image of this frame section represented by the 
still picture of 1 by which selection was made [ above-mentioned ] The step which displays serially on 
the above-mentioned panorama image display window the panorama image by which creation was 
carried out [ above-mentioned ] 

[Claim 3] The record medium which recorded the program which performs the image display approach 
according to claim 1 or 2 characterized by providing the following The step which computes further the 
information which shows the moving trucking of the photography view at the time of photoing the 
above-mentioned frame used for creating this still picture about each of the above 1 or two or more still 
pictures to the above-mentioned control section using this frame in addition to the above-mentioned step 
The step which displays the information which shows this moving trucking on the above-mentioned 
display 

[Claim 4] The record medium which recorded the program which performs the image display approach 
according to claim 1 to 3 characterized by providing the following The step which computes further the 
information which shows the transit time of the photography view at the time of photoing the above- 
mentioned frame used for creating this still picture about each of the above 1 or two or more still 
pictures to the above-mentioned control section using this frame in addition to the above-mentioned step 
The step which displays the information which shows this transit time on the above-mentioned display 
[Claim 5] Display Image input means The means which sets up the dynamic-image viewing window 
which is the image synthesizer unit equipped with the above, and indicates the frame which constitutes 
the dynamic image inputted from this input means by sequential, and is displayed on this display, A 
means to carry out sequential composition of the frame which constitutes this dynamic image, and to 
create a panorama image, The means which sets up the panorama image display window which displays 
this panorama image serially, and is displayed on this display, This panorama image by both every 
changing point of this dynamic image, and both [ either or ] for every assignment unit A break, The icon 
image display window which indicates the image of 1 or two or more break eyes of this break by list as 
1 or two or more still pictures is set up, and it is characterized by having the means displayed on this 
display. 
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The outputs of camera operation information acquisition unit, the image processing 
information acquisition unit and the picture taking state information acquisition 
unit are supplied to the representation image extraction information evaluation 
unit (6) . The evaluation unit extracts the representation image which fully 
expresses the contents of the moving image taken automatically. The picture taking 
state information memory stores the picture taking state information that has to be 
processed during reception of a sensor signal. 
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(54) ANIMATION IMAGE RECORDING MEDIUM, STILL IMAGE EXTRACTING DEVICE, 
ANIMATION IMAGE RECORDER AND STILL IMAGE AUTOMATIC EXTRACTING METHOD 

(57)Abstract: 

PURPOSE: To extract a representative image 
expressing sufficiently a content of the animation image 
as a still image automatically by providing a 
representative image extracting information evaluation 
section to the device so as to extract the image based on 
image extract information. 
CONSTITUTION: A camera operation information 
acquisition section 2, an image processing information 
acquisition section 3 and an image pickup state 
information acquisition section 4 respectively read out 
camera operation information, image processing 
information and image pickup information from a 
reproduced signal inputted by a reproduction signal input 
section 1 . Then a representative image extracting 

information evaluation section 6 detects video recording start operation information included 
in the camera operation information to evaluate whether or not each image is proper to a 
representative image of a cut image as to each image in the same cut image till the 
succeeding video recording start operation information is detected. As a result, a 
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and stores the received image to a representative image storage section 8 and then the 
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CLAIMS 



[Claim(s)] 

[Claim 1] The camera actuation information storage field where the camera actuation information that 
the photography person operated the camera when photoing a dynamic image was memorized, The 
dynamic-image record medium characterized by having at least one of the photography status 
information storage regions where the photography status information under photography which 
processed the signal from a sensor and was obtained was remembered to be the image-processing 
information storage field where the image-processing information which processed the picturized image 
and was acquired was memorized. 

[Claim 2] A camera actuation information acquisition means to incorporate the camera actuation 
information that the photography person operated the camera when photoing a dynamic image, An 
image-processing information acquisition means to incorporate the image-processing information which 
processed the picturized image and was acquired, A photography status information acquisition means 
to incorporate the photography status information under photography which processed the signal from a 
sensor and was obtained, It carries out based on at least one of the camera actuation information from 
said camera actuation information acquisition means, the image-processing information from said 
image-processing information acquisition means, and the photography status information from said 
photography status information acquisition means. The static-image extractor characterized by having a 
static-image extract means to extract the static image of at least one sheet out of the dynamic image 
photoed after a photography person does photography initiation actuation before carrying out 
photography termination actuation. 

[Claim 3] A camera actuation information acquisition means to incorporate the camera actuation 
information that the photography person operated the camera when photoing a dynamic image, An 
image-processing information acquisition means to incorporate the image-processing information which 
processed the picturized image and was acquired, A photography status information acquisition means 
to incorporate the photography status information under photography which processed the signal from a 
sensor and was obtained, It carries out based on at least one of the camera actuation information from 
said camera actuation information acquisition means, the image-processing information from said 
image-processing information acquisition means, and the photography status information from said 
photography status information acquisition means. While recording a static-image extract means to 
extract the static image of at least one sheet out of the dynamic image photoed after a photography 
person does photography initiation actuation before carrying out photography termination actuation, and 
the dynamic image photoed with image pick-up equipment on a dynamic-image record medium The 
dynamic-image recording device characterized by having a static-image information record means to 
record the information on the static image extracted with said static-image extract means. 
[Claim 4] In order to extract the static image of at least one sheet out of the dynamic image photoed after 
a photography person does photography initiation actuation before carrying out photography termination 
actuation At least one of the photography status information under photography which processed the 
signal from image-processing information and a sensor which processed camera actuation information, 
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such as a zoom of image pick-up equipment and photography initiation actuation, and the picturized 
image, and was acquired, and was obtained is considered as an input. The static-image automatic 
extracting approach characterized by calculating the evaluation value about each image based on static- 
image extract knowledge, and extracting an image with a high evaluation value. 

[Claim 5] In order to extract the static image of at least one sheet out of the dynamic image photoed after 
a photography person does photography initiation actuation before carrying out photography termination 
actuation At least one of the photography status information under photography which processed the 
signal from image-processing information and a sensor which processed the camera actuation 
information on image pick-up equipment and the picturized image, and was acquired, and was obtained 
is considered as an input. The static-image automatic extracting approach that said camera actuation 
information, said image-processing information, or said photography status information is characterized 
by extracting the image with which were satisfied of the extraction condition. 

[Claim 6] In order to extract the static image of at least one sheet out of the dynamic image photoed after 
a photography person does photography initiation actuation before carrying out photography termination 
actuation At least one of the photography status information under photography which processed the 
signal from image-processing information and a sensor which processed camera actuation information, 
such as a zoom of image pick-up equipment, and the picturized image, and was acquired from 
photography initiation actuation to the image after fixed time amount progress, and was obtained is 
considered as an input. The static-image automatic extracting approach according to claim 5 that said 
camera actuation information, said image-processing information, or said photography status 
information is characterized by extracting the image with which were satisfied of the extraction 
condition. 

[Claim 7] In order to extract the static image of at least one sheet out of the dynamic image photoed after 
a photography person does photography initiation actuation before carrying out photography termination 
actuation As opposed to the image photoed after the number of images which multiplied the number of 
the images photoed from photography initiation actuation before photography termination actuation by 
the rate of a constant ratio At least one of the photography status information under photography which 
processed the signal from image-processing information and a sensor which processed camera actuation 
information, such as a zoom of image pick-up equipment, and the picturized image, and was acquired, 
and was obtained is considered as an input. The static-image automatic extracting approach according to 
claim 5 that said camera actuation information, said image-processing information, or said photography 
status information is characterized by extracting the image with which were satisfied of the extraction 
condition. 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Industrial Application] This inventions are the static-image extractor which extracts a typical image 
automatically and the dynamic-image record medium used for it, and a thing about a dynamic-image 
recording device and the static-image automatic extracting approach further out of the dynamic image 
photoed with the video camera etc. 
[0002] 

[Description of the Prior Art] As a Prior art, there is a thing of recording the information for managing 
the recorded dynamic image on a video tape with a dynamic image, with VTR. as an example - VISS 
(VHS Index Search System) ******** „ - lt explains. With VISS, it is developed in order to perform 
high-speed search in VTR of a VHS method. The control truck which records the VISS signal for 
performing this high-speed search in addition to the video track which records the usual image 
information exists in a video tape. This VISS signal is automatically recorded on a control truck, when 
beginning to record image information on videotape on a video tape. Moreover, a VISS signal is also 
recordable to the scene which a user wants to see. Thus, rapid-traverse playback called an intro search 
can be performed using the VISS signal recorded on the video tape. If a VISS signal is found during a 
rapid traverse, an intro search will change only a certain time amount into a playback condition, and will 
repeat actuation of fast forwarding again after that, to a end of tape. 
[0003] 

[Problem(s) to be Solved by the Invention] However, since the VISS signal automatically recorded in 
die conventional VTR is recorded when beginning to record image information on videotape on a video 
tape, when the image of a part to which the VISS signal is attached like an intro search is extracted, the 
extracted image does not fiilly express the contents of the dynamic image. Moreover, if it is going to 
give a VISS signal to the part which fully expresses the contents of the dynamic image, a user has to 
carry out manually and it will take huge time and effort. 

[0004] This invention aims at the static-image extractor which extracts automatically the representation 
image which fully expresses the contents of the dynamic image and the dynamic-image record medium 
used for it, and offering a dynamic-image recording device and the static-image automatic extracting 
approach further in view of this point. 
[0005] 

[Means for Solving the Problem] In order to attain the above-mentioned purpose, the dynamic-image 
record medium of this invention The camera actuation information storage field where the camera 
actuation information that the photography person operated the camera when photoing a dynamic image 
was memorized, It is the configuration equipped with at least one of the photography status information 
storage regions where the photography status information under photography which processed the signal 
from a sensor and was obtained was remembered to be the image-processing information storage field 
where the image-processing information which processed the picturized image and was acquired was 
memorized. 



http://www4.ipdl.ncipi.go.jp/cgi-bin/tran_web_cgi_ejje 



12/20/05 



«JP,08-009314,A [DETAILED DESCRIPTION] 



Page 2 of 13 



[0006] Moreover, a camera actuation information acquisition means to incorporate the camera actuation 
information that the photography person operated the camera when the static-image extractor of this 
invention photoed a dynamic image, An image-processing information acquisition means to incorporate 
the image-processing information which processed the picturized image and was acquired, A 
photography status information acquisition means to incorporate the photography status information 
under photography which processed the signal from a sensor and was obtained, It carries out based on at 
least one of the camera actuation information from said camera actuation information acquisition means, 
the image-processing information from said image-processing information acquisition means, and the 
photography status information from said photography status information acquisition means. It has a 
static-image extract means to extract the static image of at least one sheet out of the dynamic image 
photoed after a photography person does photography initiation actuation before carrying out 
photography termination actuation. 

[0007] Furthermore, the dynamic-image recording device of this invention is equipped with a static- 
image information record means to record the information on the static image extracted with said static- 
image extract means while it records the dynamic image photoed with image pick-up equipment on a 
dynamic-image record medium. 
[0008] 

[Function] It sets in the above configurations, and it evaluates, for example based on photography status 
information, such as a pan, and an evaluation value is high or the image which is obtained by 
performing camera actuation information, such as a zoom of the photography person at the time of 
photography, and an image processing, which is obtained from image processing information, such as a 
focus, the dependability of an iris, a location of a photographic subject, and states of being of an 
obstruction, and a sensor and with which are satisfied of certain conditions is extracted out of a dynamic 
image, for example. By this, the static image extracted becomes what fully expressed the contents of the 
dynamic image. 

[0009] Moreover, it becomes possible to search a representation image easily or to output based on 
static-image information, by recording the information on the extracted static image with a dynamic 
image. 
[0010] 

[Example] Hereafter, the example of this invention is explained, referring to a drawing. Drawing 1 
shows one example of the dynamic-image record medium of this invention. Although drawing 1 shows 
a video tape as an example of a dynamic-image record medium, they may be other record media, such as 
a videodisk and an IC memory. 

[001 1] As shown in drawing 1 , corresponding to the video signal, the extract information on a 
representation image is recorded on the video tape to each field per frame with the video signal. The 
extract information on a representation image is camera actuation information, image-processing 
information, and photography status information here. In addition, it cannot be overemphasized that the 
sequence recorded on each field of the truck of a video tape is not what is restricted to the sequence 
shown in drawing. 

[0012] Camera actuation is actuation for taking out the photography effectiveness, such as the image 
transcription initiation actuation and zoom actuation at the time of taking a photograph with a video 
camera or fade, and continuous shooting, etc., image transcription initiation actuation information is the 
information stand a flag and an image transcription start point is shown, when image transcription 
initiation actuation is performed, and zoom actuation information is the information express a zoom 
scale factor. Both image transcription initiation actuation information, zoom actuation information, fade 
actuation information, and continuous-shooting actuation information are information detectable based 
on button grabbing of a video camera. 

[0013] Image-processing information is automatic or the information which human being involved and 
carried out extract processing based on the video signal picturized with the image sensor. For example, 
the frequency and magnitude of a high frequency component of a video signal for which it asked in 
order to perform focal control, or inter-frame [ which searched for the difference between the luminance 
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signal in inter-frame, or a chrominance signal ] — difference - the condition of the thing and backlight 
which extracted information, such as a location of a photographic subject field, and magnitude, from the 
value or the video signal, or a fault follow light, and contrast — they are gamma correction value, a color 
temperature, etc. further. 

[0014] Photography status information has the distance to the photographic subject by the motion 
information on cameras, such as panning which is the information which detected the photography 
condition of a camera by the sensor, for example, was detected by the angular-velocity sensor, the 
drawing opening of the lens detected by the photographic subject quantity of light and the diaphragm 
opening sensor by the quantity of light sensor or the focal distance by lens location detection, a ranging 
sensor, etc. 

[0015] By equipping a dynamic-image record medium with the above representation image extract 
information, it becomes possible to extract the representation image in a dynamic image so that 
subsequent examples may explain. 

[0016] Next, the 1st example of the static-image extractor of this invention is explained. The block 
diagram of this example is shown in drawing 2 . This example records the information for extracting a 
representation image on the dynamic-image record medium with the video signal, reads and evaluates 
the extract information on a representation image from a dynamic-image record medium, and extracts a 
typical image as a static image based on an evaluation result out of 1 cut (lump of the dynamic image 
continuously photoed after carrying out image transcription initiation actuation in a camera before 
carrying out image transcription termination actuation). 

[0017] drawing 2 — setting ~ 1 ~ the regenerative-signal input section and 2 — the camera actuation 
information acquisition section and 3 — for the video-signal acquisition section and 6, as for the 
representation image Management Department and 8, the representation image extract information- 
evaluation section and 7 are [ the image-processing information acquisition section and 4 / the 
photography status information acquisition section and 5 / the representation image storage section and 
9 ] output units. Actuation of each part in the above configuration is explained to a detail below. 
[0018] First, the information recorded on the dynamic-image record medium is reproduced and inputted 
into the regenerative-signal input section 1 . In the camera actuation information acquisition section 2, 
the image-processing information acquisition section 3, and the photography status information 
acquisition section 4, camera actuation information, image-processing information, and photography 
status information are read from the regenerative signal inputted into the regenerative-signal input 
section 1, respectively. Each coded information is decoded in this example. It evaluates [ whether it is an 
image suitable as a representation image of the cut of each image based on the information, the image- 
processing information, and the photography status information other than the image transcription 
initiation actuation information in camera actuation information, and ] to each image in the same cut 
until the representation image extract information-evaluation section 6 detects the image transcription 
initiation actuation information included in camera actuation information and it detects the following 
image transcription initiation actuation information. About the image estimated to be suitable as a 
representation image, the representation image Management Department 7 captures the image of one 
frame from the video-signal acquisition section 5, and memorizes in the representation image storage 
section 8. An output unit 9 takes out and outputs the representation image memorized by the 
representation image storage section 8, and are a display, a printer, etc. In addition, the example of the 
below-mentioned static-image automatic extracting approach explains actuation of the representation 
image extract information-evaluation section 6 to a detail. 

[0019] The information for extracting a representation image beforehand is recorded on the dynamic- 
image record medium with the video signal, and the above example explained the case where read the 
extract information on a representation image from a dynamic-image record medium, and a 
representation image was extracted. However, even when informational a part or informational all for 
extracting a representation image does not exist in a dynamic-image record medium, by processing the 
video signal recorded on the dynamic-image record medium, the information for extracting a 
representation image can be acquired and a representation image can be extracted based on the acquired 



http://www4.ipdl.ncipi.go.jp/cgi-bin/tran_web_cgi_ejje 



12/20/05 



JP,Q8-009314,A [DETAILED DESCRIPTION] 



Page 4 of 13 



information. The static-image extractor of the following example [ 2nd ] explains this to a detail. 
[0020] The 2nd example acquires all representation image extract information only from a video signal. 
The example of a configuration of the equipment which acquires representation image extract 
information from a video signal to drawing 3 is shown, drawing 3 10 — inter-frame - difference — a 
value detecting element and 1 1 - memory and 12 — a variation detecting element and 13 — a cut change 
detecting element and 14 — a camera work detecting element and 15 — a motion vector detecting 
element and 16 - the camera work parameter estimation section and 17 - for the characteristic quantity 
extract section in a field, and 20, as for a high pass filter and 22, a focal information detecting element 
and 21 are [ a photographic subject information detecting element and 18 / a motion area detecting 
element and 19 / the averaging section and 47 ] contrast information detecting elements. Actuation of 
each part in the above configuration is explained to a detail below. 

[0021] first, inter-frame difference ~ actuation of the value detecting element 10 and the cut change 
detecting element 13 is explained, inter-frame - difference - the value detecting element 10 consists of 
a variation detecting element 12 which asks for the difference of a dynamic-image signal by inter-frame 
[ which follows the memory 1 1 for delaying one dynamic-image signal ]. the difference of the inter- 
frame picture signal with which the signal which searches for the inter-frame difference which a 
dynamic image follows continues per pixel in the variation detecting element 12 using a brightness 
value, a rgb value, etc. — an operation - carrying out « the difference for every pixel ~ total of a value - 
- asking - inter-frame — difference — it outputs as a value, the cut change detecting element 13 — inter- 
frame ~ difference ~ inter-frame [ for which it asked by the value detecting element 10 ] ~ difference ~ 
threshold processing is carried out to a value, namely, inter-frame [ a predetermined threshold and inter- 
frame ] ~ difference - the comparison with a value — carrying out — inter-frame — difference - when a 
value is larger than a threshold, the contents of an image think that it is changing a lot by inter-frame [ of 
two sheets ], and it is judged that there was a cut change in the part. In a video camera, since a cut 
change arises by performing image transcription initiation actuation, image transcription initiation 
actuation can be presumed by detecting a cut change from a picture signal conversely, therefore, inter- 
frame [ which exceeds a threshold in the cut change detecting element 13 ] - difference - when a value 
is detected, image transcription initiation actuation information is outputted. 

[0022] in addition, inter-frame [ which was shown by drawing 3 ] - difference - the configuration of 
the value detecting element 10 may be an example, and other configurations as shown by drawing 4 may 
be used for it. the histogram which detects the difference in a color histogram by the color histogram 
detecting element which asks for a color histogram [ in / in 44 / one frame of a dynamic-image signal ] 
by drawing 4 , the histogram memory which memorizes the histogram which calculated 45, and inter- 
frame [ which 46 follows ] - difference - it is a detecting element. Although every pixel is not 
measured by inter-frame but the whole frame compares with the configuration shown in drawing 4 , it is 
good also as a configuration which divides a screen into two or more blocks, and asks for difference 
inter-frame in a block unit. 

[0023] Next, the camera work detecting element 14 is explained. First, actuation of the motion vector 
detecting element 15 is explained. Drawing 5 is drawing for explaining the location in the screen of the 
motion vector to detect. Drawing 5 (a) is drawing which arranged the straight line of M and N book in 
the shape of a grid to level and a perpendicular direction by the full screen, and M.N intersections show 
the location of the motion vector which should be detected. It is the lattice point (i, j) in M.N 
intersections below about the lattice point, and a call, a horizontal and the lattice point that it is 
perpendicular and are i and the j-th respectively. (1 <=i<=M, 1 <=j<=N) 
It calls. 

[0024] The motion vector in a lattice point location chooses two or more representation points in the 
circumference of each lattice point by this example, and it asks by representation point matching. 
Drawing 5 (b) is drawing expanded near the lattice point (i, j) of drawing 5 (a), and shows the physical 
relationship of the lattice point and the representation point of - (2-m+l) (2-n+l) individual of the 
circumference of it. It is a representation point (i, j, k, 1) about k and the 1-th thing, respectively to the 
level among the representation points of the following and the lattice point (i, j), and a perpendicular 
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direction. (-m<=k<=m, -n<=l<=n) 

It calls. A representation point (0 i, j, 0) is equal to the lattice point (i, j) so that drawing 5 (b) may show. 

[0025] Hereafter, how to ask for a motion vector is explained using drawing 6 which showed the 
concrete block diagram of the motion vector detecting element 15. The input of the motion vector 
detecting element 15 is a video signal, and presupposes that it is set up so that an input may be carried 
out once (r: predetermined number) at r frames. Here, the image of a certain time of day t is used as the 
image of the 0th frame, and the image of time of day (t+tau) will be henceforth called the image of a ** 
(30andtau) frame eye. 

[0026] Now, the image of the Rth frame should be inputted. In BPF23, a band pass filter lets an input 
image pass first. The value of the image after BPF processing in a coordinate location (x y) is set to I (x 
y) here. 

[0027] On the other hand, as for the representation point value storage section 24, the value of the 
representation point of the image of r-frame ago, i.e., the BPF processing back of a ** (R-r) frame eye, 
is memorized. That is, the value Y (i, j, k, 1) of a representation point (i, j, k, 1) is memorized. 
[0028] 

Y(i, j, k, 1) =1 (pos__x (i, k)) pos_y(j, 1) 1 <=i<=M, 1 <=j<=N, -m<=k<=m, x-coordinate pos_y of a - 
n<=l<=npos_x(i, k):representation point (i, j, k, 1) (i, k) : the y-coordinate matching section 25 of a 
representation point (i, j, k, 1) The value Y of the representation point r-frame before the representation 
point value storage section 24 (i, j, k, 1) is inputted for the image I after the BPF processing from BPF23 
(x y), and it asks for the motion vector in each lattice point by representation point matching. That is, 
about the lattice point (i, j), as shown in following (several 1), a motion vector (g, h) can be found by 
searching for g and h from which K becomes min within the limits of - (2andG) (2andH) (-G<=g<=G, - 
H<=h<=H). 
[0029] 
[Equation 1] 

K=II{Y(i,j,kJ)-I(pos_x(i,0)+g.pos_y(j,0)+h)} 

k 1 

[0030] The contents of the representation point value storage section 24 are updated after processing of 
the matching section 25 is completed. Specifically, the value in the representation point of the image 
after the Rth-frame BPF processing is recorded using coordinate pos_x (i, j, k, 1) of the representation 
point memorized in the representation point position-memory section 26, pos_y(i, j, k, 1) 1 <=i<=M, 1 
<=j<=N, -m<=k<=m, and -n<=l<=n. 

[0031] The image lost-motion vector, the image inputted as mentioned above and the image of r-frame 
ago, of two sheets can be searched for. 

[0032] Next, in the camera work parameter estimation section 16, how to presume a camera work 
parameter from a motion vector is explained. 

[0033] The camera work which can be presumed from a dynamic image can consider the level of a 
camera, a vertical change (panning, chill TINGU), change (zooming) of a camera field angle, change 
(tracking, booming, Dolly Inge) of the location of a horizontal, a perpendicular, and [ of a camera ] 
order, etc. By this example, since it is easy, how to presume panning, chill TINGU, and three kinds of 
actuation of zooming is explained. 

[0034] First, it considers how the point projected on the image pick-up side of a camera moves by the 
three above-mentioned kinds of camera works. Drawing 7 is drawing showing the image pick-up side of 
a camera, and the physical relationship of a photographic subject, expresses the three-dimension 
coordinate of the space of a camera with x, and (y, z), and expresses the two-dimensional coordinate on 
an image pick-up side with (X, Y). Moreover, make the location of a camera into the zero of a three- 
dimension coordinate, and let the optical axis of a camera be the z-axis. An image pick-up side is located 
in z=F (F: focal distance), and it is shown that coordinate ul= (xl, yl, zl) of the point of the arbitration 
of a photographic subject is projected on Ul= (XI, Yl) of an image pick-up side. The relation between 
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the coordinate of a photographic subject and the coordinate on an image pick-up side can be expressed 

with (several 2) here. 

[0035] 

[Equation 2] 

X1=F-X1/Z1 

Y1=F-y1/z1 

[0036] The migration on the image pick-up side of the coordinate of a photographic subject by zooming 
is first considered using the coordinate of drawing 7 . Drawing 8 (a) shows zooming which happens by 
change of a focal distance. As shown in this drawing, when a focal distance changes from F to F 1 , 
projection of the photographic subject of ul moves to U2= (X2, Y2) from Ul= (XI, Yl). However, 
(several 2) to U2 fills U2=U1 and F'/F=f-Ul . However, it is f=FVF. 

[0037] The case of panning and chill TINGU is considered using drawing 8 (b) the same way. Panning 
and chill TINGU are equal to actuation of rotating a camera about the y-axis and a x axis, respectively. 
As shown in this drawing, when a camera rotates only theta x about a x axis, the coordinate ul in the 
space of a photographic subject moves to u3. However, u3 fills (several 3). 
[0038] 
[Equation 3] 

u3=ul [10 0 
O cosex -sin0x 
0 sinex cosex 

[0039] If it assumes that the angle of rotation theta x about x is sufficiently small, the relation of X3=X1 
and Y3=Y1+ F-theta x will be drawn from the relation (several 2) to coordinate U3= on the image pick- 
up side after migration (X3, Y3). If this is generalized, in the camera actuation rotated [ as opposed to / 
both /ax axis and the y-axis ], the relation before and behind actuation of the coordinate of arbitration 
can be expressed as follows. 
[0040] U3=U1+P, however P= (px, py) 

px, py: The thing more than the rotation component of a x axis and the y-axis shows that coordinate Ul= 
before and behind camera actuation (XI, Yl) and U - (X 1 , Y') fill U -f-U+P to zooming, panning, and 
the general camera actuation that compounded chill TINGU. Henceforth, f is called a zoom element and 
P is called a rotation vector. Therefore, by searching for a zoom element and a rotation vector shows that 
the control input of a camera can be presumed. 

[0041] How to presume a zoom element and a rotation vector is explained from the motion vector for 
which below was asked by the motion vector detecting element 15. Here, the motion vector asked for 
the location (two-dimensional coordinate) by Ui, j, and the motion vector detecting element 15 is set to 
vi and j about the lattice point (i, j). When camera actuation of the zoom element f and the rotation 
vector P takes place now, the lattice point (i, j) is U'i, j(f, P) =f-Ui, and j. It should move to the location 
of +P. Therefore, what is necessary is just to ask for f and P from which error E(f, P) =sigma(U , i, j(f, P)- 
Ureali, j) 2 with the actually moved location Ureali, j=Ui, j+vi, and j become min, in order to presume f 
of the camera actuation which actually took place, and P. Since Error E is a secondary type about f and 
P, as f and P which make Error E min show it to (several 4), it is decided that it will be a meaning. 
[0042] 
[Equation 4] 

a,j<U recA iJ f Ui f j>^<Zi.jU recl i # j,liJUU>/M/N 

f= 

a.j<Ui,j,Ui,j>-<ZiJUuj.Ii.jUi f j >/M/N 



P=(ri.jU rwl i.j-f i Zi.jUi,J)/M/N 
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[0043] However, <-, -> show an inner product. Therefore, in the camera work parameter estimation 
section 16, zooming, panning, and each camera work parameter of chill TINGU can be presumed by 
inputting the motion vector detecting-element 15 lost-motion vectors vi and j and the lattice point 
locations Ui and j, and calculating f and P using (several 4). 

[0044] Next, actuation of the photographic subject information detecting element 17 is explained. The 
photographic subject information detecting element 17 extracts photographic subject information, such 
as a location of a photographic subject, and magnitude, a color, in the condition of carrying out the 
tracking of the photographic subject, with a video camera. That is, the characteristic quantity in a field is 
extracted from a motion area in the characteristic quantity extract section 19 in a field to the case where 
panning was detected by the camera work detecting element 14, and a motion area is able to be further 
detected by the motion area detecting element 18. The actuation in the motion area detecting element 18 
is explained further in full detail. 

[0045] Motion vector **** of the camera by panning detected by the motion vectors vi and j and the 
camera work detecting element 14 of the M.N lattice points in the screen detected by the motion vector 
detecting element 15 in the motion area detecting element 18 It is inputted. In the motion area detecting 
element 18, the lattice point which fills (several 5) is extracted and a different field from the motion 
vector of the camera by panning is extracted based on the connection relation of the extracted lattice 
point. 
[0046] 
[Equation 5] 

|Vi,j-Vp|>e && m Uel*»fJfe<Dflt 

[0047] In the characteristic quantity extract section 19 in a field, a center-of-gravity location, area, and a 
color are extracted from the motion area detected by the motion area detecting element 18 as 
characteristic quantity in a field. 

[0048] Next, actuation of the focal information detecting element 20 is explained. The focal information 
detecting element 20 is for detecting the pin dotage condition of an image, and is carried out based on 
the amount of the high frequency component of an image. That is, after the image has faded by the focal 
gap of a lens etc., the value of the high frequency component of a video signal becomes small. For this 
reason, it is considering as the configuration which takes out the high frequency component of an image 
with a high pass filter 21, and calculates the average of the high frequency component in the whole 
screen or the appointed field in the averaging section 22. 

[0049] Next, actuation of the contrast information detecting element 47 is explained. Detection of 
contrast information detects the bias of a histogram by the amount detecting element 49 of histogram 
biases based on the histogram for which asked for the brightness histogram of one frame and it asked by 
the brightness histogram detecting element 48, and makes this contrast information. As an example of 
the scale of the bias of a histogram, it considers as the value of the direction with much frequency of the 
low brightness part of a brightness histogram, and a high brightness part here. In addition, the scale of 
the bias of a histogram is not what was restricted to this, and is good also as an amount of gaps from 
histogram distribution of typical brightness etc. 

[0050] The information for extracting a representation image can be acquired by processing a video 
signal as mentioned above. Although this example indicated neither gamma correction value, a color 
temperature, a backlight nor a fault follow light condition, the photographic subject quantity of light, 
etc., such information can also be acquired by processing a video signal. After acquiring the information 
for extracting a representation image, about the configuration and technique of extracting a 
representation image based on the acquired information, it is the same as that of the 1st example of a 
static-image extractor, and explanation is omitted. 

[0051] Although the equipment which acquires the representation image extract information that it 
explained above acquired representation image extract information based on the video signal read from 
the dynamic-image record medium when the information for extracting a representation image did not 
exist in a dynamic-image record medium, it can also acquire representation image extract information 
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with a video camera similarly based on the video signal incorporated from the image sensor during 
photography. The configuration in this case is the same as that of drawing 3 , and although explanation 
is omitted, when representation image extract information detectable by the sensor with which the video 
camera was equipped exists, it cannot be overemphasized that it is not necessary to acquire 
representation image extract information from a video signal. Furthermore, the representation image 
extract information acquired during photography with the video camera may be recorded on a dynamic- 
image record medium with a video signal. 

[0052] Next, the configuration of the example of the dynamic-image recording device of this invention 
is shown in drawing 9 . This example is the configuration which equipped the video camera with the 
static-image extractor, determines the static image extracted as a representation image during 
photography with a video camera, and records the information on the static image extracted while 
recording a video signal on a dynamic-image record medium. The dynamic-image recording device 27 
of this example consists of the camera actuation information acquisition section 28, the image- 
processing information acquisition section 29, the photography status information acquisition section 30, 
the video-signal acquisition section 31, the representation image extract information-evaluation section 
32, the static-image information Records Department 33, and the video-signal Records Department 34 
by drawing 9 . Actuation of each part in the above configuration is explained to a detail below. 
[0053] The camera actuation information acquisition section 28 is a part which acquires information, 
such as actuation for taking out the photography effectiveness, such as the image transcription initiation 
actuation and zoom actuation at the time of taking a photograph with a video camera or fade, and 
continuous shooting. Image transcription initiation actuation information is information which sets a flag 
and shows an image transcription start point, when image transcription initiation actuation is performed, 
and zoom actuation information is the information showing a zoom scale factor. Both image 
transcription initiation actuation information, zoom actuation information, fade actuation information, 
and continuous-shooting actuation information are detected based on button grabbing of a video camera, 
inter-frame [ which searched for the difference between the frequency and the magnitude of the high 
frequency component of the video signal which the image-processing information acquisition section 29 
was the part which acquires the information which processed the video signal picturized with the image 
sensor for example, was searched for in order to perform focal control or the luminance signal in inter- 
frame, or a chrominance signal ] — difference - the condition of information, such as the location of the 
photographic subject field for which it asked from a value or a video signal, and magnitude, a backlight, 
or a fault follow light, and contrast — correction value gamma, a color temperature, etc. extract further. 
The photography status-information acquisition section 30 acquires the distance to the photographic 
subject by the motion information on cameras, such as panning which is the part which acquires the 
information which detected the photography condition of a camera by the sensor, for example, was 
detected by the angular- velocity sensor, the drawing opening of the lens detected by the photographic 
subject quantity of light and the diaphragm opening sensor by the quantity of light sensor or the focal 
distance by lens location detection, a ranging sensor, etc. 

[0054] It evaluates [ whether it is an image suitable as a representation image of the cut of each image 
based on the information, the image-processing information, and the photography status information 
other than the image transcription initiation actuation information in camera actuation information, and ] 
to each image in the same cut until the representation image extract information-evaluation section 32 
detects the image transcription initiation actuation information included in camera actuation information 
and it detects the following image transcription initiation actuation information. Static-image 
information is recorded on the static-image information record medium of a record medium 35 through 
the static-image information Records Department 33 about the image estimated to be suitable as a 
representation image. In addition, the example of the below-mentioned static-image automatic extracting 
approach explains actuation of the representation image extract information-evaluation section 32 to a 
detail. 

[0055] Below, the static-image information recorded at the static-image information Records 
Department 33 is explained further. Static-image information is the flag given to the dynamic image 
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corresponding to the storing positional information on dynamic-image record media corresponding to 
the thing which carried out the image processing, or a static image, such as the static image itself 
estimated to be suitable as a representation image by the representation image extract information- 
evaluation section 32, or contraction, or a static image. 

[0056] When static-image information is the static image itself or the reduced image, the location where 
the record locations on the dynamic-image record medium which records the dynamic image photoed 
with the camera by the video-signal acquisition section 3 1 and the video-signal Records Department 34 
differ, or the dynamic-image record medium which records a dynamic image records the image which is 
static-image information on a different record medium. For example, when a dynamic-image record 
medium is a video tape, only static-image information is recorded on the IC memory which packed only 
the image of static-image information, and recorded on a part for the head part of a tape, or the trailer of 
a tape, or it had apart from the tape. 

[0057] As for the record location on the dynamic-image record medium which records the dynamic 
image photoed with the camera by the video-signal acquisition section 3 1 and the video-signal Records 
Department 34, in the case of the storing positional information on the dynamic-image record medium 
corresponding to a static image in static-image information, the storing positional information on the 
dynamic-image record medium corresponding to the static image which is static-image information is 
recorded on a different record medium, as for a different location or the dynamic-image record medium 
which records a dynamic image. 

[0058] In the case of the flag which static-image information gave to the dynamic image corresponding 
to a static image, static-image information is recorded on the same location as the record location on the 
dynamic-image record medium which records the dynamic image photoed with the camera by the video- 
signal acquisition section 31 and the video-signal Records Department 34. That is, the flag of static- 
image information is recorded on the head part of the video signal recorded, for example per one frame. 
[0059] While recording the video signal photoed with the video camera as mentioned above on a 
dynamic-image record medium, a representation image is extracted out of the photoed dynamic image, 
and the static-image information on a representation image is recorded on a record medium. It becomes 
possible to read the static-image information recorded by this and to output a representation image to a 
display or a printer. 

[0060] Next, the example of the static-image automatic extracting approach in this invention is 
explained. The static-image automatic extracting approach is an art in the representation image extract 
information-evaluation section 6 of drawing 2 , and the representation image extract information- 
evaluation section 32 of drawing 9 . 

[0061] The static-image automatic extracting approach of this invention extracts a typical image 
automatically as a static image out of the dynamic image continuously photoed after carrying out image 
transcription initiation actuation before carrying out image transcription termination actuation. A typical 
image means the image which evaluated based on the intention of a photography person, the condition 
of the photoed image, and the condition of a photographic subject, and was selected here. 
[0062] An intention of a photography person is reflected in camera works, such as a zoom and a pan. 
That is, while zooming in, it is the case where the photographic subject currently observed exists in a 
screen, and is considered an important image. Moreover, while carrying out the pan, it is the midst 
which is moving to another scene from a certain scene, and it is thought that it is not important. 
Furthermore, when the photographic subject which is carrying out tracking even when the pan is being 
carried out exists, it is thought that it is important. Thus, it is desirable to presume an intention of a 
photography person from camera work, and to extract an important part as a representation image. 
[0063] The condition of the photoed image means conditions the condition of the image condition in 
which it faded in case focal control was not well performed at the time of photography, a fault follow 
light when iris control is unsuitable, or a backlight, when gamma amendment is still more unsuitable, the 
bad condition of contrast, a diaphragm, the condition under adjustment of a focus, etc. These image 
conditions can be judged based on the information on the focal control at the time of photography with a 
video camera, or iris control, or gamma correction value. Moreover, even if it is the case where there is 
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no information on focal control, iris control, gamma amendment, and contrast, it is possible to ask by 
processing a video signal. It is desirable to evaluate these image conditions and for an image condition 
to extract a good thing as a representation image. 

[0064] The condition of a photographic subject means conditions when a flash plate shines during the 
states of being of obstructions when people cross a camera front during conditions, such as a location of 
the photographic subject currently photoed, and magnitude, and photography, and photography, the 
condition that the spotlight is irradiated by the photographic subject, etc. It is more desirable for the one 
where area is larger to have a desirable location in the center of a camera, and for an obstruction not to 
exist about the location or magnitude of a photographic subject. Moreover, it is more desirable not to 
extract an image when a flash plate shines as a representation image. Moreover, the image with which 
the spotlight is irradiated is an attention image, and extracting as a representation image is desirable. 
[0065] Here, in the 3rd example, the photographic subject information detecting element 17 of drawing 
3 explains the detection approach of the location of a photographic subject, or area, moreover, the 
detection approach of a flash plate or an obstruction — being related — inter-frame [ of drawing 3 ] ~ 
difference - inter-frame [ for which it asked by the value detecting element 10 ] ~ difference — it is 
detectable based on a value, namely, a flash plate - inter-frame - difference - since a value changes 
suddenly, it is detectable based on a predetermined threshold, the time of an obstruction coming out 
from the time of an obstruction entering into a screen, and a screen, as for the case of an obstruction — 
inter- frame — difference — since a value changes — inter-frame — difference — since a value exceeds a 
predetermined threshold - the inside of predetermined time amount - again - inter-frame ~ difference - 
- when a value exceeds a threshold, it can detect under the condition that an obstruction exists in a 
screen. Moreover, the exposure of a spotlight is detectable based on the photographic subject quantity of 
light. 

[0066] Below based on the knowledge for extracting the above representation images, the extract 
technique of a concrete representation image is explained. The example of a configuration of this 
example is shown in drawing 10 . For 36, as for the gate signal generating section and 38, a weighting 
adder unit and 37 are [ the gate section and 39 ] maximum detecting elements in drawing 10 . A zoom 
scale factor and photographic subject information are inputted into the weighting adder unit 36, and 
weight is attached and added to each signal. Here, photographic subject information is the information 
acquired while carrying out the tracking of the photographic subject with the camera, and is carried out 
based on the location and magnitude of a photographic subject at the time of tracking. It is made for 
photographic subject information to become such a big value that the area of a photographic subject is 
so large that a photographic subject location is close to the core of a camera. The gate section 38 
performs ON of a switch, and OFF based on the gate signal of the gate signal generating section 37. The 
maximum detecting element 39 detects the maximum of a value inputted from the gate section 38. 
[0067] the gate signal generating section 37 - the high frequency component value of a pan signal and a 
video signal, and inter-frame ~ difference - a gate signal is generated based on a value etc. The 
generating approach of a gate signal is shown in drawing 1 1 . Drawing 1 1 (a) is a pan signal and is a 
signal with which under panning is 0 and the time of having not carried out a pan is set to 1 . It means 
that (b) is the high frequency component value of a video signal, and is in the condition in which the 
image faded, so that the value was small, (c) carries out threshold processing, makes the signal of (b) 
binary, and, in below a threshold, makes it 0. (d) - inter-frame - difference — it is a value, (e) carries 
out threshold processing of the signal of (d), in more than a threshold, it is made into 0, and after the 
signal of (d) exceeds a threshold further, when a threshold is again exceeded in predetermined time, it 
performs processing which also sets the section between zero to 0. 

[0068] namely, independent — inter-frame - difference - what the abnormalities in an image with a 
flash plate etc. generated when a value became large - judging - inter-frame -- difference ~ a value sets 
to 0 only the period which is more than a threshold, however - the case where an obstruction passes 
through a camera front as mentioned above ~ inter-frame - difference — the period when an obstruction 
exists in a screen in order that a value may take two or more peaks — inter-frame — difference — it is 
made 0 even if a value is below a threshold. A gate signal is generated by taking the AND of three 
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signals of (a), (c), and (e) made binary as mentioned above. 

[0069] By asking for the image with which an evaluation value turns into maximum with the 
configuration shown by drawing 10 as mentioned above, the image with which a zoom scale factor is 
high with an image and the photographic subject is greatly reflected to middle of the screen from from 
while removing the period which is carrying out the pan, the period when the image faded, and the 
period when a flash plate and an obstruction exist further can be extracted as a representation image. In 
addition, the whole 1 cut may detect maximum by the maximum detecting element 39, or they may be 
two or more sections in 1 cut. 

[0070] In addition, although one evaluation value is calculated from two or more inputs in drawing 10 
by the configuration of the weighting adder unit 36, the gate signal generating section 37, and the gate 
section 38, it is not what was restricted to this configuration, and the thing based on the Ruhr, such as 
fuzzy reasoning, and the configuration for which it asks by the neural network are also possible. 
Furthermore, although processing of the information about gamma correction value, the photographic 
subject quantity of light, a backlight or a fault follow light condition, diaphragm opening, contrast, and a 
focal distance was not shown in the configuration of this example, these signals can be used similarly. 
Namely, in a backlight or a fault follow light condition, a gate signal may be generated while changing 
the value of gamma correction value, diaphragm opening, or a focal distance, and so that it may not 
extract as a representation image. Moreover, it detects that the spotlight is irradiated from the 
photographic subject quantity of light, and may be made to make an evaluation value high. 
[0071] Other examples of a configuration of the static-image automatic extracting approach are shown 
in drawing 12 . this example — a zoom scale factor and inter-frame — difference — it is the configuration 
of calculating the maximum of a comprehensive evaluation value which computed the evaluation value 
from input, such as a value, and carried out weighting addition to each evaluation value. In drawing 12 , 
50 is the evaluation value calculation section and calculates an evaluation value from each input. For 
example, if input is a zoom scale factor, such a high evaluation value will be outputted that it is highly 
competitive, moreover, input - inter-frame - difference - if it is a value ~ difference - such a high 
evaluation value is outputted that a value is small. Moreover, if input is contrast, such a high evaluation 
value will be outputted that contrast is high. 

[0072] By asking for the image with which an evaluation value turns into maximum with the 
configuration shown by drawing 12 as mentioned above, contrast is high, not an image with which an 
image does not fade but a flash plate and an obstruction exist further but a zoom scale factor is high, and 
the image with which the photographic subject is greatly reflected to middle of the screen can be 
extracted as a representation image. 

[0073] Although a zoom scale factor and the magnitude of the photographic subject which is one of the 
photographic subject information were configurations used as independent information in case a static 
image was extracted in the above example, it is the configuration which presumes the evaluation value 
relevant to the relative magnitude of a photographic subject from a zoom scale factor and photographic 
subject distance by the static-image automatic extracting approach of this example, and carries out 
automatic extracting of the static image. 

[0074] The magnitude of the photographic subject picturized with image pick-up equipment is 
proportional to a zoom scale factor, and in inverse proportion to photographic subject distance. 
Therefore, if the photographic subject under photography is the same, the relative magnitude of the 
picturized photographic subject can be presumed from a zoom scale factor and photographic subject 
distance. 

[0075] The configuration of the static-image automatic extracting approach of this example is shown in 
drawing 13 . 5 1 is the evaluation value calculation section in drawing 13 , and 5 la is the evaluation 
value calculation section which outputs the evaluation value relevant to the relative magnitude of a 
photographic subject by considering a zoom scale factor and photographic subject distance as an input. 
Hereafter, it explains in more detail about actuation of evaluation value calculation section 51a. 
[0076] It is Dmax about the maximum distance of target within the limits [ maximum / of a now and 
zoom scale factor ] as Zmax and a photographic subject distance. It carries out. The minimum distance 
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of target within the limits as a photographic subject distance at this time is Dmax/Zmax. It becomes. If 
the zoom scale factor z and the photographic subject distance d are inputted into evaluation value 
calculation section 51a here, the evaluation value s will be calculated based on s=z-Dmax/d. 
[0077] In addition, d>=Dmax It solves and they are d=Dmax and d<=Dmax/Zmax. It solves and is 
d=Dmax/Zmax. Thus, the calculated evaluation value s turns into a value proportional to the magnitude 
of the picturized photographic subject. 

[0078] Based on the evaluation value calculated in the evaluation value calculation section 51 as 
mentioned above, the method of extracting a representation image from the weighting adder unit 36 and 
the maximum detecting element 39 is a last example, and is as having already stated, and explanation is 
omitted, in addition, the input used in order to extract a representation image by this example ~ except 
for a zoom scale factor and photographic subject distance ~ contrast and inter-frame — difference — 
although it is only a value, the input of others, such as a high frequency component, may be used. 
[0079] moreover, the approach of carrying out direct detection of the detection of photographic subject 
distance using ranging sensors, such as an infrared sensor and a supersonic sensor, — or there is the 
approach of calculating and computing photographic subject distance based on the focal lens location at 
the time of a focus. 

[0080] Although it evaluates to all the dynamic images photoed after a photography person does 
photography initiation actuation before carrying out photography termination actuation and a 
representation image is extracting, in the configuration of the example of the static-image automatic- 
extracting approach which explained above, the configuration of extracting considering the image at the 
time of carrying out evaluation from the image after predetermined-time progress after the photography 
person carried out photography initiation actuation, and being satisfied in predetermined conditions as a 
representation image is used. Below, the example of this configuration is explained. 
[0081] The configuration of this example is shown in drawing 14 . As for a timer and 41, in drawing 14 , 
40 is [ the gate section and 42 ] the evaluation sections. A timer 40 measures the elapsed time after 
photography initiation actuation is performed, and when fixed time amount has passed since 
photography initiation, it generates a gate signal so that the gate of the gate section 41 may be opened, 
the high frequency component value of the focus in which the evaluation section 42 passed the gate 
section 41, and inter-frame « difference - it evaluates whether the value has satisfied conditions, more 
than a threshold predetermined [ conditions / in the evaluation section 42 ] in the high frequency 
component value of a focus — it is -- and inter-frame - difference ~ values are the conditions of being 
below a predetermined threshold. When satisfied with the evaluation section 42 of conditions, 
evaluation after it is stopped, and the static image at the time of conditions being satisfied is extracted as 
a representation image, in addition, the signal used for evaluation in this example - the high frequency 
component value of a focus, and inter-frame - difference — although it is only two of values, other 
signals, such as a pan and a zoom, may be used. 

[0082] The configuration of example with the still more nearly another static-image automatic extracting 
approach is explained. The configuration of this example is shown in drawing 15 , the same number is 
attached to the same thing as drawing 1414 , and explanation is omitted. This example performs 
evaluation from the number of images which multiplied the number of the images photoed from 
photography initiation actuation before photography termination actuation by the rate of a constant ratio, 
and extracts the image at the time of satisfying predetermined conditions as a representation image. The 
configuration of drawing 15 estimates the image after an intermediate frame to the image photoed from 
photography initiation actuation before photography termination actuation. For this reason, in the 
intermediate-frame detecting element 43, when the intermediate frame of the frame photoed by 
photography termination actuation from photography initiation actuation is detected and an intermediate 
frame is detected, a gate signal is generated so that the gate of the gate section 41 may be opened. 
Actuation of the gate section 41 and the evaluation section 42 is the same as the configuration of 
drawing 14 , and explanation is omitted. 
[0083] 

[Effect of the Invention] As explained above, the dynamic-image record medium of this invention 
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becomes possible [ extracting easily the representation image in the dynamic image recorded on the 
dynamic-image record medium based on image extract information ] by having image extract 
information. 

[0084] Moreover, the static-image extractor of this invention becomes possible [ extracting the static 
image which fully expresses the contents of the dynamic image out of a dynamic image based on image 
extract information as a representation image ], and can grasp the contents of the dynamic image in a 
short time. 

[0085] Furthermore, the dynamic-image recording device of this invention becomes possible [ reading 
the recorded static-image information and outputting a representation image to a high speed at a display 
or a printer ] by extracting a representation image out of the photoed dynamic image, and recording the 
static-image information on a representation image on a record medium while recording the video signal 
photoed with the camera on a dynamic-image record medium. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] Drawing showing the dynamic-image record medium of one example of this invention 
[Drawing 2] The block diagram showing the configuration of the static-image extractor of the 1st 
example of this invention 

[Drawing 3] The block diagram showing the configuration of the representation image information 
extractor in the static-image extractor of the 2nd example of this invention 
[Drawing 4] inter-frame [ in the 2nd example ] — difference - the block diagram showing other 
configurations of a value detecting element 

[Drawing 5] (a) is drawing showing the lattice point which detects the motion vector in the motion 
vector detecting element in the static-image automatic extracting equipment of the 2nd example, 
(b) is drawing expanded near the lattice point (i, j) of drawing 5 (a). 

[Drawing 6] The block diagram showing the configuration of the motion vector detecting element in the 
2nd example 

[Drawing 7] Drawing showing the image pick-up side of a camera, and the physical relationship of a 
photographic subject 

[Drawing 8] (a) is drawing showing the image pick-up side of the camera in zooming, and the physical 
relationship of a photographic subject. 

(b) is drawing showing the image pick-up side of the camera in chill TINGU, and the physical 
relationship of a photographic subject. 

[Drawing 9] The block diagram showing the configuration of the dynamic-image recording apparatus of 
the example of this invention 

[Drawing 10] The block diagram showing the configuration of the representation image extract 
information-evaluation section of the static-image automatic extracting approach of the 1st example of 
this invention 

[Drawing 11] (a) - (e) is drawing showing actuation of the gate signal generating section of the static- 
image automatic extracting approach of the 1st example of this invention. 
[Drawing 12] The block diagram showing the configuration of the representation image extract 
information-evaluation section of the static-image automatic extracting approach of the 2nd example of 
this invention 

[Drawing 13] The block diagram showing the configuration of the representation image extract 
information-evaluation section of the static-image automatic extracting approach of the 3rd example of 
this invention 

[Drawing 14] The block diagram showing the configuration of the representation image extract 
information-evaluation section of the static-image automatic extracting approach of the 4th example of 
this invention 

[Drawing 15] The block diagram showing the configuration of the representation image extract 
information-evaluation section of the static-image automatic extracting approach of the 5th example of 
this invention 
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[Description of Notations] 

1 Regenerative-Signal Input Section 

2 Camera Actuation Information Acquisition Section 

3 Image-Processing Information Acquisition Section 

4 Photography Status Information Acquisition Section 

5 Video-Signal Acquisition Section 

6 Representation Image Extract Information-Evaluation Section 

7 Representation Image Management Department 

8 Representation Image Storage Section 

9 Output Unit 

28 Camera Actuation Information Acquisition Section 

29 Image-Processing Information Acquisition Section 

30 Photography Status Information Acquisition Section 

31 Video-Signal Acquisition Section 

32 Representation Image Extract Information-Evaluation Section 

33 Static-Image Information Records Department 

36 Weighting Adder Unit 

37 Gate Signal Generating Section 

38 Gate Section 

39 Maximum Detecting Element 

40 Timer 

41 Gate Section 

42 Evaluation Section 

43 Intermediate-Frame Detecting Element 

50 Evaluation Value Calculation Section 

5 1 Evaluation Value Calculation Section 
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DRAWINGS 



[Drawing 1] 
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CLAIMS 



[Claim(s)] 

[Claim 1] In the image reader equipped with the page turning-over equipment which turns over pages, 
such as books, automatically An image pick-up means to picturize a manuscript and to obtain image 
data, and the 1st concentration data detection means which detects concentration data from the image 
data of the 1st manuscript page picturized by the above-mentioned image pick-up means, The 1st 
concentration data storage means which memorizes the concentration data detected by the concentration 
data detection means of the above 1st, The 2nd concentration data detection means which detects 
concentration data from the image data of the 2nd manuscript page picturized by the above-mentioned 
image pick-up means, The 2nd concentration data storage means which memorizes the concentration 
data detected by the concentration data detection means of the above 2nd, The image reader which 
compares the concentration data of the 1st manuscript page with the concentration data of the 2nd 
manuscript page, and is characterized by having a decision means to judge it as a page turning-over error 
when the 2nd concentration data is the same as the 1st concentration data. 

[Claim 2] Concentration data are an image reader according to claim 1 which is the number of pixels 
with the concentration exceeding a predetermined threshold. 

[Claim 3] Concentration data are an image reader according to claim 1 which is the average 
concentration of a manuscript page. 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the image reader equipped with the page turning-over 

equipment which turns over pages, such as books, automatically. 

[0002] 

[Description of the Prior Art] Although the image reader which reads a manuscript in the condition of 
having opened wide upward is considered as the configuration whose operator operates page turning- 
over actuation of a manuscript etc. and the reading rate serves as a machine of medium-speed extent 
conventionally, in special use, such as input vicarious execution business, implementation of the 
equipment in which high-speed photography is more possible is desired. Furthermore, in addition to 
improvement in the speed of an input, the automation is needed in order to aim at expansion to 
applications, such as media conversion from a book manuscript to electronic intelligence, in the future. 
For example, it is expected that needs, such as power feed (page turning over of books) equipment of a 
manuscript and automatic-exchange equipment (book changer), are also produced. As an automatic 
manuscript feed gear in the conventional manuscript reader, by photography termination of a front 
manuscript, paper is automatically fed to the following manuscript, it has the completion of discharge of 
a copy paper, and there are some which count photography number of sheets. Moreover, in automatic 
manuscript feed gears, such as a scanner, it has the completion of discharge of a manuscript and there 
are some which count photography number of sheets. Moreover, in OCR equipment, the focus of image 
data is extracted and there are some which recognize the difference in a page. 
[0003] 

[Problem(s) to be Solved by the Invention] However, there were the following troubles in the above 
conventional image readers. When the delivery mistake of a form arises, there is a possibility of 
photoing the same page twice. Moreover, with a book manuscript, although migration of a sheet was 
detected and the completion of discharge of a manuscript is recognized in the automatic manuscript feed 
gear corresponding to a sheet manuscript, since one side files and is being fixed, migration of a page 
does not take place. Therefore, the error detection of page turning-over actuation is difficult. Moreover, 
since it is thin compared with a sheet form, the error of turning-over actuation tends to produce the form 
of a book manuscript. Furthermore, since processing of the error detection of page turning-over 
actuation is complicated and there is much amount of data to deal with, it is fit for neither improvement 
in the speed nor a cost cut. This invention is made in order to solve the trouble mentioned above, and it 
aims at offering the image reader which can prevent a high speed and mistake of being able to detect the 
error of page turning over certainly and photoing the same page repeatedly, without adding new 
detection equipment by distinguishing the difference in a page using a small number of parameter used 
for an image processing. 
[0004] 

[Means for Solving the Problem] In the image reader equipped with the page turning-over equipment 
with which this invention turns over pages, such as books, automatically in order to attain the above- 
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mentioned purpose An image pick-up means to picturize a manuscript and to obtain image data, and the 
1st concentration data detection means which detects concentration data from the image data of the 1st 
manuscript page picturized by the above-mentioned image pick-up means, The 1st concentration data 
storage means which memorizes the concentration data detected by the concentration data detection 
means of the above 1st, The 2nd concentration data detection means which detects concentration data 
from the image data of the 2nd manuscript page picturized by the above-mentioned image pick-up 
means, The 2nd concentration data storage means which memorizes the concentration data detected by 
the concentration data detection means of the above 2nd, The concentration data of the 1st manuscript 
page are compared with the concentration data of the 2nd manuscript page, and when the 2nd 
concentration data is the same as the 1st concentration data, it has a decision means to judge it as a page 
turning-over error. Moreover, the number of pixels with the concentration exceeding a predetermined 
threshold can be used for the above-mentioned concentration data. Moreover, the average concentration 
of a manuscript page can be used for the above-mentioned concentration data. 
[0005] In the above-mentioned configuration, the 1st concentration data detection means detects 
concentration data from the image data of the 1st manuscript page picturized by the image pick-up 
means, and the detected concentration data are memorized by the 1st concentration data storage means. 
The 2nd concentration data detection means detects concentration data from the image data of the 2nd 
manuscript page picturized by the image pick-up means, and the detected concentration data are 
memorized by the 2nd concentration data storage means. A decision means compares the concentration 
data of the 1st manuscript page with the concentration data of the 2nd manuscript page, and when these 
are the same, it judges them to be a page turning-over error. Thereby, when the page turning-over error 
has arisen, what it overlaps and outputs the reading data of the manuscript of the same page accidentally 
is lost. 
[0006] 

[Embodiment of the Invention] Hereafter, one example which materialized this invention is explained 
with reference to a drawing. Drawing 1 shows the whole image reader configuration. In this drawing, 
the image reader 1 is equipped with the manuscript base 3 where the manuscripts 2, such as books and a 
file, are opened wide and placed upward, and the image pick-up camera section 5 which reads a 
manuscript by the scan of the image pick-up sensor 4 which becomes with a CCD line sensor etc. is 
formed above this manuscript base 3. Moreover, the lighting section 6 which illuminates a manuscript, 
and the control unit 7 which sets up image reading conditions etc. are formed in the back side upper part 
of the manuscript base 3. The start key 8 which starts reading actuation is formed in the manuscript base 
3, and the reflective mirror 1 1 and the projection lens 12 driven with the lens driving gear which is not 
illustrated are formed in the camera head section 5. Moreover, the side face of a manuscript is projected 
on the image pick-up sensor 4, and the ranging mirror 14 which detects manuscript height from the 
projection image is formed in the method of the back of the manuscript base 3. Moreover, on the 
manuscript base 3, the movable bookholder 16 holding the right-and-left edge of a manuscript 2 is 
formed, and the skip equipment 1 8 which turns over the page of a manuscript 2 is formed further. 
[0007] In the above-mentioned configuration, after becoming a manuscript image and changing an 
optical path by the reflective mirror 1 1 in the image pick-up camera section 5, image formation of the 
illumination light reflected in respect of the manuscript 2 is carried out on the image pick-up sensor 4 
with the projection lens 12, and it is changed into an electrical signal here. The two-dimensional image 
of a manuscript is obtained by scanning the image pick-up sensor 4 in the direction of vertical scanning 
(longitudinal direction shown in drawing 1 by the arrow head). 

[0008] Drawing 2 shows the outline configuration which looked at this equipment from the front, the 
image pick-up camera section 5 is the optical system for carrying out image formation of the manuscript 
image to an image sensor, and consists of a taking lens 12 driven by the motor for (automatic focuses 
AF), and an image pick-up sensor 4 which a manuscript image scans in the focal plane by which image 
formation is carried out in the direction of vertical scanning (said — the direction shown by the arrow 
head by 2). Some image pick-up sensors 4 read the image of the end shape of the manuscript reflected to 
the ranging mirror 14. When the manuscript 2 laid on the manuscript base 3 is closed at the end on the 
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other hand and opened right and left, for example, it is books, a file, etc. used as the shape of the 
cylindrical surface at which each page turned spatially. The ranging mirror 14 serves as the stopper for 
manuscript positioning extended to a longitudinal direction by the back side of the manuscript base 3, to 
the front face of the manuscript base 3, at the include angle of 45 degrees, inclines and is installed. A 
manuscript 2 is aligned by applying the upper limit of a manuscript 2 to the lower limit section of this 
ranging mirror 14. Moreover, the manuscript base 3 has performed coloring deeper than the natural 
complexion concentration of a manuscript. In addition, in drawing 2 , the reflective mirror 1 1 has 
excluded illustration. 

[0009] Drawing 3 expresses the brightness histogram created from a certain image data of one line of 
the manuscript 2 stored temporarily in memory 36. The output (brightness) of an image sensor was 
taken along the axis of abscissa, and the number of pixels in the brightness (frequency) is taken along 
the axis of ordinate. It asks for the frequency of a peak by the side with brightness high among 
histograms, and let the brightness value of the lower one which has the frequency of the one half of the 
frequency be the manuscript substrate brightness of the Rhine. It considers as the threshold of the 
brightness which should judge the brightness value which deducted constant value from this substrate 
brightness to be the alphabetic character section. It asks for the alphabetic character frequency of one 
line by counting the number of pixels with brightness lower than an alphabetic character brightness 
threshold. Moreover, the concentration data concerning the average concentration of a manuscript page 
according to claim 3 compute the sum density (frequency) of each Rhine from the brightness histogram 
of drawing 3 , perform this about all reading Rhine, and should just ask for the average concentration of 
all reading Rhine. 

[0010] Drawing 4 shows the detail configuration of skip (turning over) equipment 18. Skip equipment 
18 has the delivery roller 19 which motorised is carried out and turns over a page, it goes up and down 
with a disjunction means 20 by which this roller 19 consists of a rack and a pinion, and books and 
attachment and detachment of it are enabled, and migration to the longitudinal direction of a skip of it is 
enabled by the migration means 22 which is guided at the guide member 21 and changes with a wire and 
a pulley. The detection sensor 23 which grows into the migration direction edge of the delivery roller 19 
with a limit switch etc. is arranged, and migration of the delivery roller 19 is detected. If skip actuation 
is explained, the outside approach of the page which the delivery roller 19 tends to descend and open 
wide and is going to turn over the books manuscript 2 will be contacted, and the rotation drive of this 
roller 19 will be carried out. Thereby, the middle neighborhood of a page is raised. From the delivery 
roller 19 moving leftward [ illustration ] with the migration means 22 in this condition, as for that page, 
the binding section is rotated at the supporting point, and, thereby, a page is turned over. The delivery 
roller 19 will be returned to the original location by the migration means 22 based on the detection 
signal, if having moved to the page edge is detected by the detection sensor 23. 
[001 1] Drawing 5 is the block diagram of the control circuit in the image reader 1. Equipment 1 is 
equipped with the alphabetic character frequency detecting element 32, the picture signal processing 
section 33, etc. which detect alphabetic character frequency as the image pick-up sensor 4 which 
changes by CPU31, CCD, etc. which control each part of equipment, and concentration data. CPU31 
controls the scanning means 34, in order to read a manuscript by optical scan, and it performs 
preliminary scan and this scanning actuation. After A/D conversion of the picture signal read by the 
image pick-up sensor 4 at the time of a preliminary scan is carried out by A/D converter 35, it is stored 
temporarily in memory 36 per one line. CPU31 controls and reads the alphabetic character frequency 
detecting element 32, and computes alphabetic character frequency etc. based on a picture signal, and 
memory 37 is made to memorize it. The detail of calculation of alphabetic character frequency is 
mentioned later. CPU31 controls the picture signal processing section 33 based on such information, 
performs various kinds of image processings, such as concentration conversion and variable power 
processing, and outputs image data to the printer which is an external device. Moreover, if there is no 
difference of the alphabetic character frequency of each computed continuous page, CPU31 will judge 
with the skip error having arisen, and will perform error processing. 

[0012] Drawing 6 is a flow chart which shows the outline of actuation of the image reader 1 . Hereafter, 
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reading actuation of the manuscript by equipment 1 is explained using this drawing. Here, the case 
where continuation reading is performed is shown, turning over a page for the page which the book 
manuscript followed. When the start of photography actuation is directed (#1), CPU3 1 performs 
scanning actuation (#2) and makes memory 36 memorize the image data for one line (#3). Furthermore, 
alphabetic character frequency is made to detect and memorize from this image data (#4). If it is not 
scanning termination (it is NO at #5), it will return to #2 and next Rhine will be scanned. If the scan for 
1 page finishes, the alphabetic character frequency for 1 page will be added together (#6). Next, carry 
out the pressure welding of the delivery roller 19 of skip equipment 18 to manuscript space for a skip, 
and it is made to rotate (#7), and further, if the delivery roller 19 is moved (#8) and a page edge is 
detected (it is YES at #9), the same actuation as the 1st above-mentioned page will be performed about 
the 2nd page (#10-#14). 

[0013] Then, it is judged as that by which the page was normally turned over when it investigated 
whether a difference would be in a total value with an alphabetic character frequency [ each ] of page 
[ 2nd ] with the 1st page (#15) and was, the alphabetic character frequency in memory is updated (#16), 
pagination is counted up (#17), and it investigates whether it is photography termination (#19), and if it 
is not termination, if it is return and termination, processing will be finished to #7. Moreover, in the 
above-mentioned #15, if there is no alphabetic character frequency difference, it will be judged as what 
the skip error produced, a delivery error message will be performed (#18), and it will return to #7. the 
time of photography termination — the alphabetic character frequency deletion in memory - it carries 
out. What is necessary is just to set up suitably the difference of the alphabetic character frequency 
which judges whether it is a skip error. 

[0014] By the way, when the copy tariff etc. is managed based on photography pagination, it is 
necessary to count only a part for the page which the skip was carried out normally and photoed. Then, 
only when the skip is performed normally, he is trying for CPU3 1 to output a count-up signal in the 
above-mentioned #17. 

[0015] Other examples of the approach of detecting whether the skip having been performed normally 

are explained below. In order to detect whether the skip was performed normally, there are the detection 

approach of the amount of edges (Laplacian) and the detection approach of a brightness histogram other 

than the comparison of alphabetic character frequency that what is necessary is to evaluate the 

description of an image and just to express. The amount of edges is what took the difference of an 

attention pixel and a circumference pixel twice, and by the alphabetic character image, and few values 

are taken with a photograph. A filter as shown in the following table is multiplied by image data, and, 

specifically, the result is added together. 

[Table 1] 

0 10 

i-4 i a: -4<Dtmifi&mm 

0 1 0 



[0016] As shown in above-mentioned drawing 5 R> 5, the brightness histogram of an image of the 
detection approach of a brightness histogram is a numeric value showing the substrate of a manuscript, 
and distribution of an alphabetic character. Then, distinction of whether the skip was performed 
normally is possible by taking the brightness histogram for one-page image data, and comparing the 
frequency in each brightness. 
[0017] 

[Effect of the Invention] A high speed and mistake of being able to detect the error of page turning over 
certainly and photoing the same page repeatedly can be prevented without adding new detection 
equipment, since it distinguished whether page turning over was normally performed using 
concentration data, such as alphabetic character frequency which is the parameter used for an image 
processing, according to the image reader applied to this invention as mentioned above. 
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DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is drawing showing the whole image reader configuration by one example of this 
invention. 

[Drawing 2] It is drawing showing the outline configuration which looked at the image reader from the 
front. 

[Drawing 3] It is drawing showing the brightness histogram created from a certain image data of one 
line of a manuscript. 

[Drawing 4] It is drawing showing the detail configuration of skip equipment. 
[Drawing 5] It is the block diagram of the control circuit in an image reader. 
[Drawing 6] It is the flow chart which shows the outline of actuation of an image reader. 
[Description of Notations] 
2 Manuscript 

4 Image Pick-up Sensor (Image Pick-up Means) 
1 8 Skip Equipment (Page Turning-over Equipment) 

31 CPU (Decision Means) 

32 Alphabetic Character Frequency Detecting Element (Frequency Detection Means) 
37 Memory (Frequency Storage Means) 
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DRAWINGS 




[Drawing 3] 
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CLAIMS 



[Claim(s)] 

[Claim 1] The image processing system characterized by to have an image acquisition means to acquire 
two or more images given from the outside, an image selection means to choose at least two images 
among two or more images acquired by said image acquisition means, and a panorama generation means 
perform splice doubling and generate a panorama static image to the image chosen by said image 
selection means. 

[Claim 2] It is the image processing system characterized by for said image acquisition means acquiring 
the image equivalent to one coma of an animation in an image processing system according to claim 1 in 
the format compressed by the interframe coding method or the coding method in a frame, and said 
image selection means choosing the image compressed by the coding method in a frame among two or 
more images acquired by said image acquisition means. 

[Claim 3] It is the image processing system characterized by for said image acquisition means acquiring 
the image equivalent to one coma of an animation, or the image of a still picture in an image processing 
system according to claim 1, and said image selection means choosing the image of a still picture among 
two or more images acquired by said image acquisition means. 

[Claim 4] It is the image processing system characterized by choosing the image acquired through said 
image acquisition means whenever the time amount said image selection means was beforehand decided 
to be in the image processing system according to claim 1 passed. 

[Claim 5] It is the image processing system with which it has a bearing-of-the-exposure-axis acquisition 
means to acquire the bearing of the exposure axis at the time of matching and this image being generated 
in an image processing system according to claim 1 by each image acquired by said image acquisition 
means, and is characterized by said image selection means choosing the image in which the direction 
where the bearing of the exposure axis acquired by said bearing-of-the-exposure-axis acquisition means 
was decided beforehand is shown. 

[Claim 6] It is the image processing system which equips each image acquired by said image acquisition 
means with a field angle acquisition means to compute or acquire the field angle of matching and this 
image, in an image processing system according to claim 5, and is characterized by said image selection 
means determining the direction of [ at the time of choosing an image ] based on the field angle 
computed or acquired with said field angle acquisition means. 

[Claim 7] As opposed to two or more images acquired through an image acquisition means to acquire 
two or more images supplied, and said image acquisition means, from the exterior So that the magnitude 
of the same object shown in each image which is the image processing system equipped with a 
panorama generation means to perform splice doubling and to generate a panorama static image, and is 
adjoined and joined together with said panorama generation means may be in agreement It is the image 
processing system characterized by having a magnitude conversion means to expand or reduce this 
image, and for said panorama generation means joining together the image expanded or reduced by said 
magnitude conversion means, and generating a panorama static image. 

[Claim 8] It is the image processing system characterized by equipping each image acquired by said 
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image acquisition means with a scale-factor acquisition means to compute or acquire the photography 
scale factor of matching and this image, in an image processing system according to claim 7, basing said 
magnitude conversion means on the photography scale factor computed or acquired by said scale-factor 
acquisition means, and expanding or reducing said image. 

[Claim 9] As opposed to two or more images acquired through an image acquisition means to acquire 
two or more images supplied, and said image acquisition means, from the exterior It is the image 
processing system equipped with a panorama generation means to perform splice doubling and to 
generate a panorama static image. A brightness detection means to detect a difference of the brightness 
the duplication part of the image joined together by said panorama generation means or near the joint, 
Based on a difference of the brightness detected by said brightness detection means, it has a brightness 
conversion means to change into the brightness which was able to determine the brightness of said 
image beforehand. Said panorama generation means The image processing system characterized by 
joining together the image from which brightness was changed and generating a panorama static image 
with said brightness conversion means. 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention] This invention relates to the image processing system which joins two or more 

images together and generates a panorama static image. 

[0002] 

[Description of the Prior Art] Conventionally, there is a panorama static-image listing device which is 
carried by JP,6-284321,A, JP,8-307770,A, etc. as equipment which generates the panorama static image 
of one sheet from the dynamic image with which the wide range photographic subject was photoed by 
panning. The technique of using for JP,6-284321,A the motion vector of the image equivalent to each 
coma of the dynamic image photoed with the camera one apparatus video tape recorder etc., and 
performing superposition of a continuous image is carried. 

[0003] Moreover, the technique which amends distortion of the image produced in change of the 
distance to the photographic subject by aberration etc., and generates a panorama static image is carried 
by JP,8-307770,A. 
[0004] 

[Problem(s) to be Solved by the Invention] Although an operator can direct with these conventional 
techniques about the range which generates a panorama static image from the photoed dynamic image, a 
panorama static image is generated using all the images of such within the limits. Therefore, when 
panning was performed at a low speed, the duplication part increased and there was a problem that 
processing effectiveness was bad. 

[0005] Moreover, since it was not assumed that photography conditions, such as exposure and a focal 
distance, change during animation photography, there was a case where the case where splice doubling 
cannot be performed, and a joint were conspicuous. Then, invention according to claim 7 to 9 aims at 
offering the image processing system which generates a panorama static image with a sufficient 
precision for the purpose of invention according to claim 1 to 6 offering the image processing system 
which generates a panorama static image promptly. 
[0006] 

[Means for Solving the Problem] Drawing 1 is the principle block diagram of invention according to 
claim 1 to 4. Invention according to claim 1 is characterized by to have an image acquisition means 10 
acquire two or more images given from the outside, an image selection means 1 1 choose at least two 
images among two or more images acquired by the image acquisition means 10, and a panorama 
generation means 12 perform splice doubling and generate a panorama static image to the image chosen 
by the image selection means 1 1 . 

[0007] Invention according to claim 2 acquires the image with which the image acquisition means 10 is 
equivalent to one coma of an animation in an image processing system according to claim 1 in the 
format compressed by the interframe coding method or the coding method in a frame, and it is 
characterized by the image selection means 1 1 choosing the image compressed by the coding method in 
a frame among two or more images acquired by the image acquisition means 10. 
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[0008] Invention according to claim 3 acquires the image with which the image acquisition means 10 is 
equivalent to one coma of an animation, or the image of a still picture in an image processing system 
according to claim 1, and it is characterized by the image selection means 1 1 choosing the image of a 
still picture among two or more images acquired by the image acquisition means 10. Invention 
according to claim 4 is characterized by the image selection means 1 1 choosing the image acquired 
through the image acquisition means 10 whenever the time amount on which it decided beforehand 
passed in an image processing system according to claim 1 . 

[0009] Drawing 2 is the principle block diagram of invention according to claim 5. Invention according 
to claim 5 is equipped with a bearing-of-the-exposure-axis acquisition means 14 to acquire the bearing 
of the exposure axis at the time of matching and this image being generated in an image processing 
system according to claim 1 by each image acquired by the image acquisition means 10, and the image 
selection means 1 1 is characterized by the bearing of the exposure axis acquired by the bearing-of-the- 
exposure-axis acquisition means 14 choosing the image in which the direction decided beforehand is 
shown. 

[0010] Drawing 3 is the principle block diagram of invention according to claim 6. Invention according 
to claim 6 equips each image acquired by the image acquisition means 10 with a field angle acquisition 
means 16 to compute or acquire the field angle of matching and this image, in an image processing 
system according to claim 5, and the image selection means 1 1 is characterized by determining the 
direction of [ at the time of choosing an image ] based on the field angle computed or acquired with the 
field angle acquisition means 16. 

[001 1] Drawing 4 is the principle block diagram of invention according to claim 7. An image acquisition 
means 20 to acquire two or more images with which invention according to claim 7 is supplied from the 
outside, It is the image processing system equipped with a panorama generation means 21 to perform 
splice doubling and to generate a panorama static image to two or more images acquired through the 
image acquisition means 20. So that the magnitude of the same object shown in each image adjoined 
and joined together with the panorama generation means 21 may be in agreement It is characterized by 
having a magnitude conversion means 22 to expand or reduce this image, and for the panorama 
generation means 21 joining together the image expanded or reduced by the magnitude conversion 
means 22, and generating a panorama static image. 

[0012] Drawing 5 is the principle block diagram of invention according to claim 8. Invention according 
to claim 8 equips each image acquired by the image acquisition means 20 with a scale-factor acquisition 
means 24 to compute or acquire the photography scale factor of matching and this image, in an image 
processing system according to claim 7, the magnitude conversion means 22 is based on the 
photography scale factor computed or acquired by the scale-factor acquisition means 24, and it is 
characterized by to expand or reduce an image. 

[0013] Drawing 6 is the principle block diagram of invention according to claim 9. An image acquisition 
means 30 to acquire two or more images with which invention according to claim 9 is supplied from the 
outside, It is the image processing system equipped with a panorama generation means 3 1 to perform 
splice doubling and to generate a panorama static image to two or more images acquired through the 
image acquisition means 30. A brightness detection means 32 to detect a difference of the brightness the 
duplication part of the image joined together by the panorama generation means 3 1 or near the joint, 
Based on a difference of the brightness detected by the brightness detection means 32, it has a brightness 
conversion means 33 to change into the brightness which was able to determine the brightness of an 
image beforehand. The panorama generation means 31 It is characterized by joining together the image 
from which brightness was changed and generating a panorama static image with the brightness 
conversion means 33. 

[0014] (Operation) In the image processing system in connection with invention according to claim 1, 
the image acquisition means 10 acquires two or more images given from the outside, and the image 
selection means 1 1 chooses at least two images among two or more images acquired by the image 
acquisition means 10. Moreover, it is inherited to the image chosen by the image selection means 1 1, the 
panorama generation means 12 permitting duplication, performs doubling, and generates a panorama 
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static image. 

[0015] Therefore, as compared with the case where all the images acquired through the image 
acquisition means 10 are joined together, the processing effectiveness at the time of generating a 
panorama static image can be improved. With the image processing system in connection with invention 
according to claim 2, the image acquisition means 10 acquires the image equivalent to one coma of an 
animation in the format compressed by the interframe coding method or the coding method in a frame. 
[0016] Moreover, the image selection means 1 1 chooses the image compressed by the coding method in 
a frame among two or more images acquired by the image acquisition means 10. It is inherited to the 
image chosen in this way, the^panorama generation means 12 permitting duplication, performs doubling, 
and generates a panorama static image. Therefore, since degradation of the image by motion 
compensation processing etc. is controlled as compared with the case where the image by which 
interframe coding was carried out is chosen, a high-definition panorama static image is generable. 
[0017] Moreover, since it can elongate easily, without referring to other frames, when only the image 
encoded in the frame generates a panorama static image, the processing effectiveness of the image 
encoded in the frame to elongating the image by which interframe coding was carried out using the 
frame of order at the time of elongation can improve, and it can generate a panorama static image 
promptly. 

[0018] In the image processing system in connection with invention according to claim 3, the image 
acquisition means 10 acquires the image equivalent to one coma of an animation, or the image of a still 
picture. Moreover, the image selection means 1 1 chooses the image of a still picture among two or more 
images acquired by the image acquisition means 10. Therefore, since splice doubling can be performed 
using the still picture photoed at spacing beforehand decided during photography of an animation, the 
still picture which the operator photoed intentionally, a high-definition panorama static image is 
generable. 

[0019] In the image processing system in connection with invention according to claim 4, the image 
selection means 1 1 chooses the image acquired through the image acquisition means 10, whenever the 
time amount on which it decided beforehand passes. That is, since the image which should perform 
splice doubling can be chosen at spacing based on the time amount on which it decided beforehand, a 
panorama static image is certainly [ promptly and ] generable. 

[0020] In the image processing system in connection with invention according to claim 5, the bearing- 
of-the-exposure-axis acquisition means 14 acquires the bearing of the exposure axis at the time of 
matching and an image being generated by each image acquired by the image acquisition means 10. 
Moreover, the image selection means 1 1 chooses the image in which the direction where the bearing of 
the exposure axis acquired by the bearing-of-the-exposure-axis acquisition means 14 was decided 
beforehand is shown. That is, a panorama static image is promptly generable by choosing the image in 
which the bearing of the exposure axis decided beforehand is shown from the image which the 
photography person changed bearing of the exposure axis, and photoed the perimeter. 
[0021] In the image processing system in connection with invention according to claim 6, the field angle 
acquisition means 16 computes or acquires the field angle of matching and this image in each image 
acquired by the image acquisition means 10. Moreover, the image selection means 1 1 determines the 
direction of [ at the time of choosing an image ] based on the field angle computed or acquired with the 
field angle acquisition means 16. That is, since the image which performs matching and splice doubling 
for the field angle and bearing of the exposure axis of the image which the photography person changed 
bearing of the exposure axis, and photoed the perimeter is chosen, the precision to join together can 
improve and a panorama static image can be generated certainly. 

[0022] In the image processing system in connection with invention according to claim 7, the image 
acquisition means 20 acquires two or more images supplied from the outside, and it is inherited to two 
or more images acquired through the image acquisition means 20, the panorama generation means 21 
permitting duplication, performs doubling, and generates a panorama static image. The magnitude 
conversion means 22 expands or reduces an image so that the magnitude of the same object shown in 
each image adjoined and joined together with the panorama generation means 21 may be in agreement. 
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[0023] Therefore, even if it is the case where the magnitude of the object shown in the image which 
should perform splice doubling differs, a panorama static image is certainly generable. In the image 
processing system in connection with invention according to claim 8, the scale-factor acquisition means 
24 computes or acquires matching and the photography scale factor of an image in each image acquired 
by the image acquisition means 20. Moreover, with the scale-factor acquisition means 24, the magnitude 
conversion means 22 is based on the photography scale factor computed or acquired, and expands or 
reduces an image. 

[0024] That is, the magnitude of the same object shown in two or more images which should perform 
splice doubling can be made easily in agreement by acquiring a photography scale factor. Therefore, 
even if it is the case where the photography scale factor of the image which performs splice doubling 
changes, a panorama static image is certainly generable. In the image processing system in connection 
with invention according to claim 9, the image acquisition means 30 acquires two or more images 
supplied from the outside. It is inherited to two or more images acquired through the image acquisition 
means 30, the panorama generation means 31 permitting duplication, performs doubling, and generates a 
panorama static image. 

[0025] Moreover, the brightness detection means 32 detects a difference of the brightness the 
duplication part of the image joined together by the panorama generation means 3 1 or near the joint, and 
adjusts the brightness conversion means 33 to the brightness which was able to determine the brightness 
of an image beforehand based on a difference of the brightness detected by the brightness detection 
means 32. Therefore, even if it is the case where the brightness the duplication part of the image joined 
together or near the joint differs, a panorama static image can be generated, without highlighting a joint. 
[0026] 

[Embodiment of the Invention] Hereafter, based on a drawing, a detail is explained about the operation 
gestalt of this invention. In addition, suppose that it explains using the electronic camera which is one 
gestalt of an image processing system as an operation gestalt corresponding to invention according to 
claim 1 to 9. 

[0027] (First operation gestalt) Drawing 7 is drawing showing the appearance of an electronic camera, 
and drawing 8 is the functional block diagram of the operation gestalt corresponding to invention given 
in claims 1 and 2. In drawing 8 R> 8, as for a control section 40, the image pick-up section 44, a disk 
drive 46, the display process section 48, a monitor 50, a touch panel 52, memory 54, a compression zone 
56, the elongation section 58, the motion vector detecting element 60, and the correlation detecting 
element 62 are connected through a system bus 42. Moreover, a disk drive 46 is loaded with a magnetic 
disk 64. 

[0028] In addition, about the correspondence relation of the principle block diagram and this operation 
gestalt which are shown in drawing 1 , the image acquisition means 10 corresponds to a disk drive 46 
and a magnetic disk 64, the image selection means 1 1 corresponds to the function which chooses the 
image data of a control section 40, and the panorama generation means 12 corresponds to the function 
which joins the image of memory 54, the motion vector detecting element 60, the correlation detecting 
element 62, and a control section 40 together. 

[0029] Moreover, with this operation gestalt, since it is easy, it is compressed in the format which the 
dynamic-image data crossed to 360 perimeters photoed by panning by the photography person apply to 
MPEG, and suppose that it is beforehand stored in a magnetic disk 64 as a dynamic-image file. Here, a 
dynamic-image file decides to consist of a header field which consists of incidental information, such as 
photography time and the total frame number, an image field which consists of dynamic-image data, and 
a thumbnail field which consists of image data of the thumbnail image mentioned later. Moreover, the 
static-image file stored in a magnetic disk 64 with a dynamic-image file decides to consist of a header 
field and an image field. 

[0030] Drawing 9 is the operation flow chart of the operation gestalt corresponding to invention given in 
claims 1 and 2. Hereafter, with reference to drawing 8 and 9, actuation of the operation gestalt 
corresponding to invention given in claims 1 and 2 is explained. 

[0031] A control section 40 will order the display-processing section 48 the display of the actuation 
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screen containing a thumbnail image as shown in drawing 1010 , if a main power supply is switched on 
( drawing 9 SI). In addition, in drawing 10 R> 0, a rewind button 100, a fast forward button 101, an 
earth switch 102, the panorama carbon button 103, the reverse playback carbon button 104, the playback 
carbon button 105, etc. show a manual operation button, and "Thumbnail A", "Thumbnail B", etc. which 
were surrounded in the rectangle show the condition that the thumbnail image is displayed. Moreover, it 
is shown that the thumbnail images surrounded in the double line or highlights are processing objects, 
such as playback, and it is shown that a thick thumbnail display is an animation. 
[0032] The display process section 48 reads the image data of a thumbnail image from the thumbnail 
field of each dynamic-image file in a magnetic disk 64 through a disk drive 46, and stores it in the 
location where the internal frame memory (not shown) was beforehand decided with the image data 
which shows thickness. Moreover, the display process section 48 reads static-image data from the static- 
image file in a magnetic disk 64 through a disk drive 46, performs pixel consistency conversion, and 
stores it in a frame memory as a thumbnail image. 

[0033] Furthermore, the display process section 48 carries out overlay processing of the image data 
stored in the frame memory, and the image data equivalent to a manual operation button which was 
mentioned above, and gives them to a monitor 50. In a monitor 50, the image data given by doing in this 
way is displayed as an actuation screen. 

[0034] A control section 40 judges whether generation of a panorama static image was directed in the 
condition that the actuation screen is displayed, by external actuation received through a touch panel 52 
( drawing 9 S2). That is, if a thumbnail image is chosen (here, suppose that "Thumbnail D" was chosen) 
and external actuation of the panorama carbon button 103 is carried out, it will recognize that generation 
of a panorama static image was directed, and the dynamic-image file corresponding to a thumbnail 
image will be opened through a disk drive 46. 

[0035] Moreover, a control section 40 reads the image data (image data encoded in the frame) of the 
frame which is equivalent to I picture based on the GOP (Group Of Picture) structure decided 
beforehand one by one from such a dynamic-image file through a disk drive 46 ( drawing 9 R>9S3). 
Thus, it is elongated in the elongation section 58 and sequential storing of the read image data is carried 
out at memory 54. 

[0036] A control section 40 detects the motion vector of the image of the frame unit stored in memory 
54 through the motion vector detecting element 60 ( drawing 9 S4). Moreover, a control section 40 
computes the inter-frame amount of gaps based on the motion vector detected in this way, and asks for 
the duplication part of the image of each frame ( drawing 9 S5). Furthermore, a control section 40 
searches for correlation for every Rhine of the perpendicular direction in the duplication part of each 
frame through the correlation detecting element 62 ( drawing 9 S6). 

[0037] Here, suppose that correlation with the pixel for one line of the perpendicular direction in the 
center section of the duplication part of one frame and the pixel of each Rhine in near the center section 
of the duplication part of the frame of another side is searched for between two adjoining frames. A 
control section 40 joins an image together in Rhine which shows the strongest correlation among Rhine 
which did in this way and searched for correlation ( drawing 9 S7). 

[0038] For example, as shown in drawing 1 1 , when the image data A and B of a frame unit is stored in 
memory 54, the image data which is equivalent even to a right end from "Rhine which shows the 
strongest correlation" of image data B is laid on top of image data A which copied image data A to other 
fields, and was copied in this way. until it recognizes that judged whether splice doubling of the image 
of all the frames in memory 54 completed the control section 40 ( drawing 9 R>9S8), and generation of 
a panorama static image was completed by such judgment — joining together (processing after drawing 
9 S4) - it repeats. In addition, it joins together in Rhine which shows the strongest correlation in a 
duplication part also about the head frame and the last frame which were read from the dynamic-image 



[0039] Thus, the generated panorama static image is compressed through a compression zone 56, and is 
written in a magnetic disk 64 as an image file through a disk drive 46 ( drawing 9 S9). Namely, a 
panorama static image can be generated with a promptly and sufficient precision by choosing only the 



file. 
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frame encoded in the frame, without choosing the frame by which predicting coding was carried out 
according to this operation gestalt. 

[0040] By the way, the thumbnail image in which the image file equivalent to a panorama static image is 
shown may be displayed in the configuration which curved like the "thumbnail E" shown in drawing 
12 . That is, a thin thumbnail image shows a static image in a rectangle, the thumbnail image which is 
thick in a rectangle shows a dynamic image, and the thumbnail image of the curved configuration shows 
a panorama static image. 

[0041] Therefore, an operator can recognize promptly the configuration (a static image / dynamic 
image / panorama static image) of the data of the image file stored in the magnetic disk 64 by displaying 
in this way. Moreover, when some panorama static images are displayed within the limit of the double 
line as it is shown in drawing 13 , when such "a thumbnail E" is chosen and external actuation of the 
panorama carbon button 103 is carried out, and external actuation of a rewind button 100 and the fast 
forward button 101 (or the reverse playback carbon button 104 and the playback carbon button 105) is 
carried out, the panorama static image may be indicated by scrolling. 

[0042] Here, a display and scrolling display of some panorama static images can be performed, without 
changing hardware by applying the operating member of the existing electronic camera as a panorama 
carbon button 103. In addition, it is also possible in the case of a scrolling display to display the 
termination of image data and the tip of image data continuously. 

[0043] (Second operation gestalt) Drawing 14 is the operation flow chart of the operation gestalt 
corresponding to invention given in claims 1 and 3. In addition, the description of this operation gestalt 
is in the procedure of processing of a control section 40, and about the configuration of hardware, since 
it is the same as the functional block diagram of the operation gestalt shown in drawing 7 , illustration is 
omitted here. 

[0044] Moreover, let the electronic camera with which this operation gestalt is applied be the thing 
which can photo a still picture during photography of an animation. In addition, when such photography 
is performed, static-image data are compressed in the format applied to JPEG, and are stored in a 
magnetic disk 64 as a static-image file. On the other hand, dynamic-image data are divided when a static 
image is photoed, and they are stored in a magnetic disk 64 as two or more dynamic-image files 
compressed in the format applied to MPEG. 

[0045] Here, these static-image files and a dynamic-image file decide to be provided for an operator as a 
single dynamic-image file, and in the header field of each file, they decide that the purport to which 
these files are making bidirectional chain structure is memorized while "the start address of the file to 
precede" and "the start address of a file which follows" are stored. 

[0046] Hereafter, with reference to drawing 7 and 14, actuation of the operation gestalt corresponding to 
invention given in claims 1 and 3 is explained. A control section 40 will order the display-processing 
section 48 the display of the actuation screen which contains a thumbnail image like the operation 
gestalt mentioned above, if a main power supply is switched on ( drawing 14 SI). The display process 
section 48 stores in an internal frame memory the thumbnail image in which the dynamic-image file and 
static-image file in a magnetic disk 64 are shown like the operation gestalt mentioned above. Moreover, 
the display process section 48 carries out overlay processing of the image data stored in the frame 
memory in this way, and the image data equivalent to the manual operation button shown in drawing 
10 , and gives them to a monitor 50. 

[0047] In a monitor 50, the image data given by doing in this way is displayed as an actuation screen. A 
control section 40 judges whether generation of a panorama static image was directed like the operation 
gestalt mentioned above in the condition that the actuation screen is displayed, by external actuation 
received through a touch panel 52 ( drawing 14 S2). 

[0048] A control section 40 will open the dynamic-image file which recognizes that generation of a 
panorama static image was directed and is equivalent to the selected thumbnail image in it, if a 
thumbnail image is chosen and external actuation of the panorama carbon button 103 is carried out like 
the operation gestalt mentioned above. In addition, it supposes that "Thumbnail D" was chosen as this 
operation gestalt showed to drawing 10 R> 0, and suppose that it is stored in the format which the image 
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data equivalent to the dynamic image crossed to 360 perimeters photoed as a dynamic-image file 
equivalent to "Thumbnail D" by panning by the photography person and the static image photoed during 
photography of such a dynamic image mentioned above. 

[0049] It judges whether the control section 40 consists of multiple files to which the dynamic-image 
file makes bidirectional chain structure with reference to the header field of the dynamic-image file 
equivalent to "Thumbnail D." That is, it judges whether static-image data are recorded with dynamic- 
image data ( drawing 14 S3). A control section 40 reads static-image data from a static-image file by 
carrying out sequential reference of the "start address of the file which follows" stored in the header 
field of each file by such judgment through the disk drive 46 when it had been recognized that static- 
image data are recorded (YES side of drawing 14 S3) ( drawing 14 S4). 

[0050] Thus, it is elongated in the elongation section 58 and sequential storing of the read image data is 
carried out at memory 54. In addition, a control section 40 decides to read the image data encoded in the 
frame like the operation gestalt mentioned above one by one, when it has been recognized that static- 
image data are not recorded (NO side of drawing 14 S3) (processing after drawing 9 S3). 
[0051] A control section 40 detects the motion vector between each static image through the motion 
vector detecting element 60 in the condition of static-image data being read from a static-image file, and 
being stored in memory 54 ( drawing 14 S5). Moreover, based on the motion vector detected in this 
way, a control section 40 computes the amount of gaps between static images, and asks for the 
duplication part of each static image ( drawing 14 S6). 

[0052] Furthermore, a control section 40 searches for correlation through the correlation detecting 
element 62 for every Rhine of the perpendicular direction in the duplication part of each static image 
( drawing 14 S7). Here, suppose that correlation with the pixel for one line of the perpendicular direction 
in the center section of the duplication part of one static image and the pixel of each Rhine in near the 
center section of the duplication part of the static image of another side is searched for between two 
adjoining static images. 

[0053] Moreover, a control section 40 is Rhine which shows the strongest correlation among Rhine 
which searched for correlation, and performs splice doubling of an image like the operation gestalt 
mentioned above ( drawing 14 S8). Furthermore, a control section 40 judges whether splice doubling of 
all the images in memory 54 was completed ( drawing 14 S9), and it repeats splice doubling until it 
recognizes that generation of a panorama static image was completed by such judgment. In addition, 
about a top static image and the last top static image, image data is deleted in Rhine which shows the 
strongest correlation. 

[0054] Thus, the generated panorama static image is compressed through a compression zone 56, and is 
written in a magnetic disk 64 as an image file through a disk drive 46 ( drawing 14 S10). That is, 
according to this operation gestalt, a high-definition panorama static image is generable using the still 
picture photoed during photography of an animation. 

[0055] In addition, although the electronic camera with which photography of a still picture is performed 
is applied during photography of an animation with this operation gestalt, when a photography person 
does external actuation of the panorama carbon button 103 during photography of an animation, it has 
the function which can choose the frame for generating a panorama static image positively, and the 
electronic camera which records the frame chosen in this way as a still picture may be applied, for 
example. 

[0056] Moreover, when external actuation of the panorama carbon button 103 is carried out during 
photography of an animation in this way, splice doubling can be smoothly performed by changing 
photography mode automatically and picturizing by keeping constant drawing of a photography scale 
factor, exposure level, and a lens etc. Furthermore, although two or more dynamic-image files and 
static-image files are making bidirectional chain structure with this operation gestalt, as long as this 
invention is not limited to the image data of such structure but can deal with dynamic-image data and 
static-image data as continuous image data, it may be the image data stored in what kind of format. 
[0057] (Third operation gestalt) Drawing 15 is the operation flow chart of the operation gestalt 
corresponding to invention given in claims 1 and 4. In addition, the description of this operation gestalt 
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is in the procedure of processing of a control section 40, and about the configuration of hardware, since 
it is the same as the fiinctional block diagram of the operation gestalt shown in drawing 7 , illustration is 
omitted here. 

[0058] Moreover, suppose that the dynamic-image data which a photography person crosses to 360 
perimeters which took a photograph by performing panning where a zoom is kept constant are stored in 
a magnetic disk 64 as a dynamic-image file with this operation gestalt. In addition, suppose that it is 
stored in the header field of a dynamic-image file about the information on the focal distance 
corresponding to the zoom kept constant in this way. 

[0059] Furthermore, although a field angle generally shows the include angle of the direction of a 
vertical angle of a screen, since it is easy, suppose that it is explained by making the horizontal include 
angle of a screen into a field angle with this operation gestalt and the operation gestalt mentioned later. 
Hereafter, with reference to drawing 7 and 15, actuation of the operation gestalt corresponding to 
invention given in claims 1 and 4 is explained. 

[0060] A control section 40 will order the display-processing section 48 the display of the actuation 
screen which contains a thumbnail image like each operation gestalt mentioned above, if a main power 
supply is switched on ( drawing 15 SI). The display process section 48 stores the thumbnail image of 
the dynamic-image file in a magnetic disk 64, and a static-image file in an internal frame memory like 
each operation gestalt mentioned above. Moreover, the display process section 48 carries out overlay 
processing of the image data stored in the frame memory in this way, and the image data equivalent to 
the manual operation button shown in drawing 10 , and gives them to a monitor 50. 
[0061] In a monitor 50, the image data given by doing in this way is displayed as an actuation screen. A 
control section 40 judges whether generation of a panorama static image was directed like each 
operation gestalt mentioned above in the condition that the actuation screen is displayed, by external 
actuation received through a touch panel 52 ( drawing 15 S2). 

[0062] A control section 40 will open the dynamic-image file equivalent to the selected thumbnail 
image, if it recognizes that generation of a panorama static image was directed like each operation 
gestalt mentioned above, the information on the header field of the dynamic-image file which opened 
the control section 40 in this way to a focal distance ~ acquiring — the field angle of the frame unit of 
dynamic-image data — asking ( drawing 15 S3) — the total image transcription time amount is acquired 
from the header field ( drawing 15 R>5 S4). 

[0063] Here, although a control section 40 chooses the frame used in case a panorama static image is 
generated, it decides to choose a frame from the head of a dynamic-image file for every fixed time 
interval with this operation gestalt. In addition, such a time interval is called "frame sampling interval." 
[0064] A control section 40 computes a "frame sampling interval" based on a field angle and the total 
image transcription time amount ( drawing 15 S5). For example, what is necessary is for a zoom to be 
maintained at a wide angle side and for a field angle to be 90 degrees, and just to make a "frame 
sampling interval" into 3 seconds, if it assumes generating a panorama static image by ten frames in 
consideration of duplication of each frame when the total image transcription time amount is 30 seconds. 

[0065] Moreover, a control section 40 reads image data from a dynamic-image file in a frame unit based 
on the computed "frame sampling interval" ( drawing 15 S6). Thus, it is elongated in the elongation 
section 58 and sequential storing of the read image data is carried out at memory 54. Furthermore, a 
control section 40 detects the motion vector of the image of the frame unit stored in memory 54 through 
the motion vector detecting element 60 like the operation gestalt corresponding to invention given in 
claims 1 and 2 ( drawing 15 S7). A control section 40 computes the inter-frame amount of gaps based on 
the motion vector detected in this way, and asks for the duplication part of the image of each frame 
( drawing 15 S8V 

[0066] Moreover, a control section 40 searches for correlation for every Rhine of the perpendicular 
direction in the duplication part of each frame through the correlation detecting element 62 ( drawing 15 
S9). Furthermore, a control section 40 performs splice doubling of an image like each operation gestalt 
which is Rhine which shows the strongest correlation among Rhine which searched for correlation, and 
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was mentioned above ( drawing 15 S10). Furthermore, a control section 40 judges whether splice 
doubling of all the images in memory 54 was completed ( drawing 15 SI 1), is inherited like each 
operation gestalt mentioned above, and repeats doubling until it recognizes that generation of a 
panorama static image was completed by such judgment. 

[0067] Thus, the generated panorama static image is compressed through a compression zone 56, and is 
written in a magnetic disk 64 as an image file through a disk drive 46 ( drawing 15 SI 2). 
[0068] That is, since it can choose easily with the "frame sampling interval" which was able to 
determine beforehand the image data which should perform splice doubling according to this operation 
gestalt, a panorama static image is promptly generable. In addition, it is possible to perform splice 
doubling smoothly, since the Bure is detectable with a motion vector even if an electronic camera is the 
Bure ** case up and down during photography of an animation with each operation gestalt mentioned 
above. 

[0069] Moreover, although a panorama static image is generated with each operation gestalt mentioned 
above from the dynamic image obtained by the photography crossed to 360 perimeters, a panorama 
static image may be generated from the dynamic image with which a photograph may have been taken, a 
photography person moving. 

(Fourth operation gestalt) Drawing 16 is the functional block diagram of the operation gestalt 
corresponding to invention given in claims 1, 5-9. 

[0070] In drawing, the same sign is given and shown about what has a function the same as the 
functional block diagram shown in drawing 7 . A control section 70 is connected to the image pick-up 
section 72, a disk drive 46, the display process section 48, a monitor 50, a touch panel 52, memory 54, a 
compression zone 56, the elongation section 58, the correlation detecting element 62, the intensity-level 
detecting element 74, and the image-processing section 76 through a system bus 42. Moreover, a disk 
drive 46 is loaded with a magnetic disk 64, and the angular- velocity sensor 78 and the focal distance 
detecting element 80 are formed in the image pick-up section 72. 

[0071] In addition, the angular-velocity sensor 78 detects change of the bearing of the exposure axis of 
an electronic camera on the basis of bearing at the time of photography being started, and the focal 
distance detecting element 80 detects a focal distance based on the zoom actuation at the time of an 
animation being photoed by the photography person, moreover, about the correspondence relation of the 
principle block diagram and this operation gestalt which are shown in drawing 1 -6 The image 
acquisition means 10, 20, and 30 correspond to a disk drive 46 and a magnetic disk 64. The image 
selection means 1 1 corresponds to the function which chooses the image data of a control section 70. 
The panorama generation means 12, 21, and 3 1 correspond to the function which joins the image of 
memory 54, the correlation detecting element 62, and a control section 70 together. The bearing-of-the- 
exposure-axis acquisition means 14 corresponds to a disk drive 46 and a magnetic disk 64. The field 
angle acquisition means 16 corresponds to the function which computes the field angle of a disk drive 
46, a magnetic disk 64, and a control section 70. The magnitude conversion means 22 corresponds to the 
function which reduces the image data of the image-processing section 76. The scale-factor acquisition 
means 24 corresponds to the function which computes a disk drive 46, a magnetic disk 64, and the 
photography scale factor of a control section 70, the brightness detection means 32 corresponds to the 
intensity-level detecting element 74, and the brightness conversion means 33 corresponds to the 
conversion function of the intensity level of the image-processing section 76. 

[0072] Drawing 17 is the operation flow chart of the operation gestalt corresponding to invention given 
in claims 1,5-9. Hereafter, with reference to drawing 16 and 17, actuation of the operation gestalt 
corresponding to invention given in claims 1, 5-9 is explained. In addition, with this operation gestalt, 
since it is easy, it is compressed in the format which the dynamic-image data crossed to 360 perimeters 
photoed by panning by the photography person apply to MPEG, and suppose that it is beforehand stored 
in a magnetic disk 64 as a dynamic-image file. 

[0073] Moreover, suppose that it matches with each frame and the variation (henceforth "whenever 
[ angle-of-coverage ]") of the bearing of the exposure axis from a head frame and the information on a 
focal distance are stored in the header field of a dynamic-image file. A control section 70 will open the 
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dynamic-image file equivalent to the selected thumbnail image through a disk drive 46, if it recognizes 
that generation of a panorama static image was directed like each operation gestalt mentioned above. 
[0074] Image data is read from the dynamic-image file opened in this way in a frame unit, it elongates in 
the elongation section 58, and a control section 70 carries out sequential storing at memory 54 ( drawing 
17 SI). Moreover, a control section 70 reads the information on a focal distance from the header field of 
a dynamic-image file, and computes the photography scale factor (henceforth a "wide angle side 
photography scale factor") of the frame which shows the shortest focal distance ( drawing 17 S2). 
[0075] Furthermore, it is directed in the image-processing section 76 that a control section 70 changes 
the photography scale factor of each frame into a "wide angle side photography scale factor 11 ( drawing 
17 S3). The image-processing section 76 reduces image data per frame based on the ratio of the 
photography scale factor and the "wide angle side photography scale factor" which computed the 
photography scale factor with reference to the information on the focal distance corresponding to each 
frame, and were computed in this way. That is, a photography scale factor is changed into a "wide angle 
side photography scale factor" by reducing image data. 

[0076] Although the frame used in case a control section 70 generates a panorama static image here is 
chosen, since processing joined together with such selection is performed repeatedly, suppose that the 
frame used as the criteria to join together is called a "precedence frame" with this operation gestalt. 
Moreover, suppose that the frame which serves as a candidate who joins together to the "precedence 
frame" is called a "consecutiveness frame." 

[0077] That is, in an initial state, a head frame is treated as a "precedence frame." Moreover, the 
"consecutiveness frame" which processing of splice **** completed is treated as a "precedence frame" 
by the processing which is performed succeedingly and to join together. A control section 70 performs 
initialization (setup of the initial value of beta and K mentioned later) in connection with the processing 
to join together ( drawing 17 S4). 

[0078] moreover, the control section 70 — the information on a focal distance — being based — the field 
angle theta 1 of a "precedence frame" ~ computing ( drawing 17 S5) — the difference of "whenever 
[ angle-of-coverage ]" with a "precedence frame" chooses the frame (it is equivalent to a 
"consecutiveness frame") which shows beta ( drawing 17 S6). In addition, beta is a value which 
specifies the difference of "whenever [ angle-of-coverage ]", and decides that any value (here, it may be 
60 degrees) is set up in the case of initialization ( drawing 17 S4). 

[0079] A control section 70 computes the field angle theta 2 of the "consecutiveness frame" chosen in 
this way ( drawing 17 S7), and is (theta l+theta2) / 2-beta>=K between the field angle theta 1 and the 
field angles theta2 and beta. ... It judges whether the relational expression of a formula 1 is realized 
( drawing 17 S8). In addition, K is zero or more values and specifies the include angle of the duplication 
part of a "precedence frame" and a "consecutiveness frame." Here, when "/(theta l+theta2)2-beta" is less 
than zero, it is shown that a "precedence frame" and a "consecutiveness frame" do not lap. 
[0080] When a formula 1 is not realized (: which corresponds when the include angle of the case where 
a "precedence frame" and a "consecutiveness frame" do not lap, or a duplication part is under K the NO 
side of drawing 17 S8), a control section 70 substitutes beta-5 for beta ( drawing 17 S9), repeats 
processing (processing equivalent to drawing 17 S5) of selection of a frame, and performs it. That is, 
spacing of selection of the frame used for generation of a panorama static image is narrowed. 
[0081] On the other hand, when a formula 1 is realized (: which corresponds when the include angle of a 
duplication part is more than K the YES side of drawing 17 S8), a control section 70 directs the 
adjustment of the brightness of the duplication part of each frame for which asked for the duplication 
part based on "/(theta l+theta2)2-beta" ( drawing 17 S10), and it asked in this way in the intensity-level 
detecting element 74 and the image-processing section 76 ( drawing 17 SI 1). 
[0082] If the intensity-level detecting element 74 is directed in this way, it will ask for the average 
intensity level of the duplication part in each frame. Moreover, based on the difference of the average 
intensity level of the duplication part in each frame, the image-processing section 76 adjusts the 
brightness of the image data of the whole "consecutiveness frame" so that a continuity with a 
"precedence frame" may be maintained. A control section 70 will search for correlation with the pixel 
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for one line of the perpendicular direction in the center section of the duplication part of a "precedence 
frame", and the pixel of each Rhine near the center section of the duplication part of a "consecutiveness 
frame" through the correlation detecting element 62, if the brightness of image data is adjusted in this 
way ( drawing 17 S12). 

[0083] A control section 70 joins an image together in Rhine which shows the strongest correlation 
among Rhine which did in this way and searched for correlation ( drawing 17 SI 3). A control section 70 
judges whether splice doubling of the image over 360 degrees was completed ( drawing 17 R>7S14), 
and it repeats the processing (processing after drawing 17 S4) which selection of a frame including 
initialization and an image join together until it recognizes that generation of a panorama static image 
was completed by such judgment. 

[0084] Thus, the generated panorama static image is compressed through a compression zone 56, and is 
written in a magnetic disk 64 as an image file through a disk drive 46 ( drawing 17 SI 5). Therefore, 
since the image which should inherit adjusting the include angle of an inter-frame duplication part, and 
should perform doubling can be chosen certainly according to this operation gestalt, a panorama static 
image is generable with a promptly and sufficient precision. 

[0085] In addition, although the angular-velocity sensor 78 which detects "whenever [ angle-of- 
coverage ]" is formed with this operation gestalt, by forming the angular- velocity sensor which measures 
the angle of rotation over an optical axis, and measuring the posture of an electronic camera in a detail, 
Bure at the time of panning can be amended and the discontinuity of a panorama static image can be 
reduced. Moreover, although the image data crossed to 360 perimeters is stored in the magnetic disk 64 
as an image file with each operation gestalt mentioned above, this invention is applicable even if it is the 
image file which consists of image data of less than 360 degrees. 

[0086] Furthermore, although splice doubling on either side is performed based on the image crossed to 
360 perimeters and the panorama static image is generated with each operation gestalt mentioned above, 
it is not limited to such a panorama static image, but an image is joined together up and down, and this 
invention can also generate a panorama static image. 
[0087] 

[Effect of the Invention] As mentioned above, in invention according to claim 1, the processing 
effectiveness at the time of generating a panorama static image can be improved as compared with the 
case where all the images acquired through the image acquisition means are joined together. 
[0088] Moreover, in invention according to claim 2, since degradation of the image by motion 
compensation processing etc. is controlled as compared with the case where the image by which 
interframe coding was carried out is chosen, a high-definition panorama static image is generable. Since 
the processing effectiveness at the time of elongation moreover improves, a panorama static image is 
promptly generable. Furthermore, in invention according to claim 3, since splice doubling can be 
performed using the still picture photoed at spacing beforehand decided during photography of an 
animation, the still picture which the operator photoed intentionally, a high-definition panorama static 
image is generable. 

[0089] Moreover, in invention according to claim 4, since the image which should perform splice 
doubling can be chosen at spacing based on the time amount on which it decided beforehand, a 
panorama static image is certainly [ promptly and ] generable. Furthermore, in invention according to 
claim 5, a panorama static image is promptly generable by choosing the image in which the bearing of 
the exposure axis decided beforehand is shown from the image which the photography person changed 
bearing of the exposure axis, and photoed the perimeter. 

[0090] Moreover, in invention according to claim 6, since the image which performs matching and 
splice doubling for the field angle and bearing of the exposure axis of the image which the photography 
person changed bearing of the exposure axis, and photoed the perimeter is chosen, the precision to join 
together can improve and a panorama static image can be generated certainly. Furthermore, in invention 
according to claim 7, even if it is the case where the magnitude of the object shown in the image which 
should perform splice doubling differs, a panorama static image is certainly generable. 
[0091] Moreover, in invention according to claim 8, since the magnitude of the same object shown in 
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two or more images which should perform splice doubling by acquiring a photography scale factor can 
be made easily in agreement, even if it is the case where the photography scale factor of the image 
which performs splice doubling changes, a panorama static image is certainly generable. Furthermore, in 
invention according to claim 9, even if it is the case where the brightness the duplication part of the 
image joined together or near the joint differs, a panorama static image can be generated, without 
highlighting a joint. 
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* NOTICES * 

JPO and NCIPI are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 



[Brief Description of the Drawings] 

[Drawing 1] It is the principle block diagram of invention according to claim 1 to 4. 
[Drawing 2] It is the principle block diagram of invention according to claim 5. 
[Drawing 3] It is the principle block diagram of invention according to claim 6. 
[Drawing 4] It is the principle block diagram of invention according to claim 7. 
[Drawing 5] It is the principle block diagram of invention according to claim 8. 
[Drawing 6] It is the principle block diagram of invention according to claim 9. 
[Drawing 7] It is drawing showing the appearance of an electronic camera. 

[Drawing 81 It is the functional block diagram of the operation gestalt corresponding to invention given 
in claims 1 and 2. 

[Drawing 9] It is the operation flow chart of the operation gestalt corresponding to invention given in 
claims 1 and 2. 

[Drawing 10] It is drawing showing the example of a display of a monitor. 
[Drawing 11] It is drawing explaining splice doubling of an image. 
[Drawing 12] It is drawing showing the example of a display of a monitor. 
[Drawing 13] It is drawing showing the example of a display of a monitor. 

[Drawing 14] It is the operation flow chart of the operation gestalt corresponding to invention given in 
claims 1 and 3. 

[Drawing 15] It is the operation flow chart of the operation gestalt corresponding to invention given in 
claims 1 and 4. 

[Drawing 16] It is the functional block diagram of the operation gestalt corresponding to invention given 
in claims 1, 5-9. 

[Drawing 17] It is the operation flow chart of the operation gestalt corresponding to invention given in 

claims 1, 5-9. 

[Description of Notations] 

10, 20, 30 Image acquisition means 

1 1 Image Selection Means 

12, 21, 31 Panorama generation means 

14 Bearing-of-the-Exposure-Axis Acquisition Means 

16 Field Angle Acquisition Means 

22 Magnitude Conversion Means 

24 Scale-Factor Acquisition Means 

32 Brightness Detection Means 

33 Brightness Conversion Means 
40 70 Control section 

42 System Bus 

44 72 Image pick-up section 

46 Disk Drive 
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48 Display-Processing Section 

50 Monitor 

52 Touch Panel 

54 Memory 

56 Compression Zone 

58 Elongation Section 

60 Motion Vector Detecting Element 

62 Correlation Detecting Element 

64 Magnetic Disk 

74 Intensity-Level Detecting Element 

76 Image-Processing Section 

78 Angular-Velocity Sensor 

80 Focal Distance Detecting Element 

100 Rewind Button 

101 Fast Forward Button 

102 Earth Switch 

103 Panorama Carbon Button 

104 Reverse Playback Carbon Button 

105 Playback Carbon Button 



[Translation done.] 
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1 .This document has been translated by computer. So the translation may not reflect the original 
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3. In the drawings, any words are not translated. 
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[Drawing 14] 
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[Drawing 15] 
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